Standardise - data#

Each of these functions parses a specific type of data file and returns a dictionary containing the data and metadata.

Surface observations#

openghg.standardise.surface.parse_aqmesh(data_filepath, metadata_filepath, sampling_period=None, **kwargs)[source]#

Read AQMesh data files

Parameters:

data_filepath (Union[str, Path]) – Data filepath
metadata_filepath (Union[str, Path]) – Metadata filepath
sampling_period (Optional[str]) – Measurement sampling period (str)

Returns:

Dictionary of data

Return type:

dict

openghg.standardise.surface.parse_beaco2n(data_filepath, site, network, inlet, instrument='shinyei', sampling_period=None, **kwargs)[source]#

Read BEACO2N data files

Parameters:

data_filepath (Union[str, Path]) – Data filepath
site (str) – Site name
network (str) – Network name
inlet (str) – Inlet height in metres
instrument (Optional[str]) – Instrument name
sampling_period (Optional[str]) – Measurement sampling period

Returns:

Dictionary of data

Return type:

dict

openghg.standardise.surface.parse_btt(data_filepath, site='BTT', network='LGHG', inlet=None, instrument=None, update_mismatch='never')[source]#

Reads NPL data files and returns the UUIDS of the Datasources the processed data has been assigned to

Parameters:

data_filepath (Union[str, Path]) – Path of file to load
site (Optional[str]) – Site name
update_mismatch (str) –
This determines how mismatches between the internal data “attributes” and the supplied / derived “metadata” are handled. This includes the options:
- ”never” - don’t update mismatches and raise an AttrMismatchError
- ”from_source” / “attributes” - update mismatches based on input data (e.g. data attributes)
- ”from_definition” / “metadata” - update mismatches based on associated data (e.g. site_info.json)

Returns:

Dictionary of gas data

Return type:

dict

openghg.standardise.surface.parse_cranfield(data_filepath, site=None, network=None, inlet=None, instrument=None, sampling_period=None, measurement_type=None, **kwargs)[source]#

Creates a CRDS object holding data stored within Datasources

Parameters:

filepath – Path of file to load
data_filepath (Union[str, Path]) – Filepath of data to be read
site (Optional[str]) – Name of site
network (Optional[str]) – Name of network

Returns:

Dictionary of gas data

Return type:

dict

openghg.standardise.surface.parse_crds(data_filepath, site, network, inlet=None, instrument=None, sampling_period=None, measurement_type=None, drop_duplicates=True, update_mismatch='never', site_filepath=None, **kwargs)[source]#

Parses a CRDS data file and creates a dictionary of xarray Datasets ready for storage in the object store.

Parameters:

data_filepath (Union[str, Path]) – Path to file
site (str) – Three letter site code
network (str) – Network name
inlet (Optional[str]) – Inlet height
instrument (Optional[str]) – Instrument name
sampling_period (Union[str, float, int, None]) – Sampling period in seconds
measurement_type (Optional[str]) – Measurement type e.g. insitu, flask
drop_duplicates (bool) – Drop measurements at duplicate timestamps, keeping the first.
update_mismatch (str) –
This determines how mismatches between the internal data “attributes” and the supplied / derived “metadata” are handled. This includes the options:
- ”never” - don’t update mismatches and raise an AttrMismatchError
- ”from_source” / “attributes” - update mismatches based on input data (e.g. data attributes)
- ”from_definition” / “metadata” - update mismatches based on associated data (e.g. site_info.json)
site_filepath (Union[str, Path, None]) – Alternative site info file (see openghg/supplementary_data repository for format). Otherwise will use the data stored within openghg_defs/data/site_info JSON file by default.

Returns:

Dictionary of gas data

Return type:

dict

openghg.standardise.surface.parse_eurocom(data_filepath, site, sampling_period, network=None, inlet=None, instrument=None, update_mismatch='never')[source]#

Parses EUROCOM data files into a format expected by OpenGHG

Parameters:

data_filepath (Union[str, Path]) – Path of file to read
site (str) – Site code
sampling_period (str) – Sampling period in seconds
network (Optional[str]) – Network name
Inlet – Inlet height in metres
Instrument – Instrument name
update_mismatch (str) –
This determines how mismatches between the internal data “attributes” and the supplied / derived “metadata” are handled. This includes the options:
- ”never” - don’t update mismatches and raise an AttrMismatchError
- ”from_source” / “attributes” - update mismatches based on input data (e.g. data attributes)
- ”from_definition” / “metadata” - update mismatches based on associated data (e.g. site_info.json)

Returns:

Dictionary of measurement data

Return type:

dict

openghg.standardise.surface.parse_gcwerks(data_filepath, precision_filepath, site, network, inlet=None, instrument=None, sampling_period=None, measurement_type=None, update_mismatch='never', site_filepath=None)[source]#

Reads a GC data file by creating a GC object and associated datasources

Parameters:

data_filepath (Union[str, Path]) – Path of data file
precision_filepath (Union[str, Path]) – Path of precision file
site (str) – Three letter code or name for site
instrument (Optional[str]) – Instrument name
network (str) – Network name
update_mismatch (str) –
This determines how mismatches between the internal data “attributes” and the supplied / derived “metadata” are handled. This includes the options:
- ”never” - don’t update mismatches and raise an AttrMismatchError
- ”from_source” / “attributes” - update mismatches based on input data (e.g. data attributes)
- ”from_definition” / “metadata” - update mismatches based on associated data (e.g. site_info.json)
site_filepath (Union[str, Path, None]) – Alternative site info file (see openghg/supplementary_data repository for format). Otherwise will use the data stored within openghg_defs/data/site_info JSON file by default.

Returns:

Dictionary of source_name : UUIDs

Return type:

dict

openghg.standardise.surface.parse_noaa(data_filepath, site, measurement_type, inlet=None, network='NOAA', instrument=None, sampling_period=None, update_mismatch='never', site_filepath=None, **kwarg)[source]#

Read NOAA data from raw text file or ObsPack NetCDF

Parameters:

data_filepath (Union[str, Path]) – Data filepath
site (str) – Three letter site code
inlet (Optional[str]) – Inlet height (as value unit e.g. “10m”)
measurement_type (str) – One of (“flask”, “insitu”, “pfp”)
network (str) – Network, defaults to NOAA
instrument (Optional[str]) – Instrument name
sampling_period (Optional[str]) – Sampling period
update_mismatch (str) –
This determines how mismatches between the internal data attributes and the supplied / derived metadata are handled. This includes the options:
- ”never” - don’t update mismatches and raise an AttrMismatchError
- ”attributes” - update mismatches based on input attributes
- ”metadata” - update mismatches based on input metadata
site_filepath (Union[str, Path, None]) – Alternative site info file (see openghg/supplementary_data repository for format). Otherwise will use the data stored within openghg_defs/data/site_info JSON file by default.

Returns:

Dictionary of data and metadata

Return type:

dict

openghg.standardise.surface.parse_npl(data_filepath, site='NPL', network='LGHG', inlet=None, instrument=None, sampling_period=None, measurement_type=None, update_mismatch='never')[source]#

Reads NPL data files and returns the UUIDS of the Datasources the processed data has been assigned to

Parameters:

data_filepath (Union[str, Path]) – Path of file to load
site (str) – Site name
network (str) – Network, defaults to LGHG
inlet (Optional[str]) – Inlet height. Will be inferred if not specified
instrument (Optional[str]) – Instrument name
sampling_period (Optional[str]) – Sampling period
measurement_type (Optional[str]) – Type of measurement taken e.g.”flask”, “insitu”
update_mismatch (str) –
This determines how mismatches between the internal data “attributes” and the supplied / derived “metadata” are handled. This includes the options:
- ”never” - don’t update mismatches and raise an AttrMismatchError
- ”from_source” / “attributes” - update mismatches based on input data (e.g. data attributes)
- ”from_definition” / “metadata” - update mismatches based on associated data (e.g. site_info.json)

Returns:

UUIDs of Datasources data has been assigned to

Return type:

list

openghg.standardise.surface.parse_tmb(data_filepath, site='TMB', network='LGHG', inlet=None, instrument=None, sampling_period=None, measurement_type=None, update_mismatch='never', site_filepath=None, **kwargs)[source]#

Reads THAMESBARRIER data files and returns the UUIDS of the Datasources the processed data has been assigned to

Parameters:

data_filepath (Union[str, Path]) – Path of file to load
site (str) – Site name
network (str) – Network, defaults to LGHG
inlet (Optional[str]) – Inlet height. Will be inferred if not specified
instrument (Optional[str]) – Instrument name
sampling_period (Optional[str]) – Sampling period
measurement_type (Optional[str]) – Type of measurement taken e.g.”flask”, “insitu”
update_mismatch (str) –
This determines how mismatches between the internal data “attributes” and the supplied / derived “metadata” are handled. This includes the options:
- ”never” - don’t update mismatches and raise an AttrMismatchError
- ”from_source” / “attributes” - update mismatches based on input data (e.g. data attributes)
- ”from_definition” / “metadata” - update mismatches based on associated data (e.g. site_info.json)
site_filepath (Union[str, Path, None]) – Alternative site info file (see openghg/supplementary_data repository for format). Otherwise will use the data stored within openghg_defs/data/site_info JSON file by default.

Returns:

UUIDs of Datasources data has been assigned to

Return type:

list

Column data#

openghg.standardise.column.parse_openghg(data_filepath, satellite=None, domain=None, selection=None, site=None, species=None, network=None, instrument=None, platform='satellite', chunks=None, **kwargs)[source]#

Parse and extract data from pre-formatted netcdf file which already matches expected OpenGHG format.

The arguments specified below are the metadata needed to store this surface observation file within the object store. If these keywords are not included within the attributes of the netcdf file being passed then these arguments must be specified.

For column data this can either be a satellite (e.g. satellite=”GOSAT”) or a site (site=”RUN”, network=”TCCON”). Either can be specified or this function will attempt to extract this from the data file.

Parameters:

data_filepath (Union[str, Path]) – Path of observation file
satellite (Optional[str]) – Name of satellite (if relevant)
domain (Optional[str]) – For satellite only. If data has been selected on an area include the identifier name for domain covered. This can map to previously defined domains (see openghg_defs “domain_info.json” file) or a newly defined domain.
selection (Optional[str]) – For satellite only, identifier for any data selection which has been performed on satellite data. This can be based on any form of filtering, binning etc. but should be unique compared to other selections made e.g. “land”, “glint”, “upperlimit”. If not specified, domain will be used.
site (Optional[str]) – Site code/name (if relevant). Can include satellite OR site.
species (Optional[str]) – Species name or synonym e.g. “ch4”
instrument (Optional[str]) – Instrument name e.g. “TANSO-FTS”
network (Optional[str]) – Name of in-situ or satellite network e.g. “TCCON”, “GOSAT”
platform (str) – Type of platform. Should be one of: - “satellite” - “site” Note: this will be superceded if site or satellite keywords are specified.
chunks (Optional[Dict]) – Chunking schema to use when storing data. It expects a dictionary of dimension name and chunk size, for example {“time”: 100}. If None then a chunking schema will be set automatically by OpenGHG. See documentation for guidance on chunking: https://docs.openghg.org/tutorials/local/Adding_data/Adding_ancillary_data.html#chunking. To disable chunking pass in an empty dictionary.
kwargs (str) – Any additional attributes to be associated with the data.

Returns:

Dictionary of source_name : data, metadata, attributes

Return type:

Dict

Emissions / flux#

Metadata#

These ensure the metadata and attributes stored with data are correct.

openghg.standardise.meta.assign_attributes(data, site=None, network=None, sampling_period=None, update_mismatch='never', site_filepath=None, species_filepath=None)[source]#

Assign attributes to each site and species dataset. This ensures that the xarray Datasets produced are CF 1.7 compliant. Some of the attributes written to the Dataset are saved as metadata to the Datasource allowing more detailed searching of data.

If accessing underlying stored site or species definitions, this will be accessed from the openghg/supplementary_data repository by default.

Parameters:

data (Dict) – Dictionary containing data, metadata and attributes
site (Optional[str]) – Site code
sampling_period (Union[str, float, int, None]) – Number of seconds for which air sample is taken. Only for time variable attribute
network (Optional[str]) – Network name
update_mismatch (str) –
This determines how mismatches between the internal data “attributes” and the supplied / derived “metadata” are handled. This includes the options:
- ”never” - don’t update mismatches and raise an AttrMismatchError
- ”from_source” / “attributes” - update mismatches based on input data (e.g. data attributes)
- ”from_definition” / “metadata” - update mismatches based on associated data (e.g. site_info.json)
site_filepath (Union[str, Path, None]) – Alternative site info file
species_filepath (Union[str, Path, None]) – Alternative species info file

Returns:

Dictionary of combined data with correct attributes assigned to Datasets

Return type:

dict

openghg.standardise.meta.get_attributes(ds, species, site, network=None, global_attributes=None, units=None, scale=None, sampling_period=None, date_range=None, site_filepath=None, species_filepath=None)[source]#

This function writes attributes to an xarray.Dataset so that they conform with the CF Convention v1.6

Attributes of the xarray DataSet are modified, and variable names are changed

If accessing underlying stored site or species definitions, this will be accessed from the openghg/supplementary_data repository by default.

Variable naming related to species name will be defined using define_species_label() function.

Parameters:

ds (Dataset) – Should contain variables such as “ch4”, “ch4 repeatability”. Must have a “time” dimension.
species (str) – Species name. e.g. “CH4”, “HFC-134a”, “dCH4C13”
site (str) – Three-letter site code
network (Optional[str]) – Network site is associated with
global_attribuates – Dictionary containing any info you want to add to the file header (e.g. {“Contact”: “Contact_Name”})
units (Optional[str]) – This routine will try to guess the units unless this is specified. Options are in units_interpret
scale (Optional[str]) – Calibration scale for species.
sampling_period (Union[str, float, int, None]) – Number of seconds for which air sample is taken. Only for time variable attribute
date_range (Optional[List[str]]) – Start and end date for output If you only want an end date, just put a very early start date (e.g. [“1900-01-01”, “2010-01-01”])
site_filepath (Union[str, Path, None]) – Alternative site info file
species_filepath (Union[str, Path, None]) – Alternative species info file

Return type:

Dataset

openghg.standardise.meta.assign_flux_attributes(data, species=None, source=None, domain=None, units='mol/m2/s', prior_info_dict=None)[source]#

Assign attributes for the input flux dataset within dictionary based on metadata and passed arguments.

Parameters:

data (Dict) – Dictionary containing data, metadata and attributes
species (Optional[str]) – Species name
source (Optional[str]) – Source name
domain (Optional[str]) – Domain name
units (str) – Unit values for the “flux” variable. Default = “mol/m2/s”
prior_info_dict (Optional[dict]) –
Dictionary of additional ‘prior’ information about for the emissions sources. Expect this to be of the form e.g.

{“EDGAR”: {“version”: “v4.3.2”,
”raw_resolution”: “0.1 degree x 0.1 degree”, “reference”: “http://edgar.jrc.ec.europa.eu/overview.php?v=432_GHG” …},

…}

Returns:

Same format as inputted but with updated “data” component (Dataset)

Return type:

Dict

openghg.standardise.meta.define_species_label(species, species_filepath=None)[source]#

Define standardised label to use for observation datasets. This uses the data stored within openghg_defs/data/site_info JSON file by default with alternative names (‘alt’) defined within.

Formatting:

species label will be all lower case
any spaces will be replaced with underscores
if species or synonym cannot be found, species name will used
but with any hyphens taken out (see also openghg.util.clean_string function)

Note: Suggested naming for isotopologues should be d<species><isotope>, e.g. dCH4C13, or dCO2C14

Parameters:

species (str) – Species name.
species_filepath (Union[str, Path, None]) – Alternative species info file.

Returns:

Both the species label to be used exactly and the original attribute: key needed to extract additional data from the ‘site_info.json’ attributes file.

Return type:

str, str

Example

>>> define_species_label("methane")
    ("ch4", "CH4")
>>> define_species_label("radon")
    ("rn", "Rn")
>>> define_species_label("cfc-11")
    ("cfc11", "CFC11")
>>> define_species_label("CH4C13")
    ("dch4c13", "DCH4C13")

openghg.standardise.meta.metadata_default_keys()[source]#: Defines default values expected within ObsSurface metadata. :returns: keys required in metadata :rtype: list

openghg.standardise.meta.sync_surface_metadata(metadata, attributes, keys_to_add=None, update_mismatch='never')[source]#

Makes sure any duplicated keys between the metadata and attributes dictionaries match and that certain keys are present in the metadata.

Parameters:

metadata (Dict) – Dictionary of metadata
attributes (Dict) – Attributes
keys_to_add (Optional[List]) – Add these keys to the metadata, if not present, based on
Note (the attribute values.) – this skips any keys which can’t be
values. (copied from the attribute) –
update_mismatch (str) –
If case insensitive mismatch is found between an attribute and a metadata value, this determines the function behaviour. This includes the options:
- ”never” - don’t update mismatches and raise an AttrMismatchError
- ”from_source” / “attributes” - update mismatches based on input attributes
- ”from_definition” / “metadata” - update mismatches based on input metadata

Returns:

Aligned metadata, attributes

Return type:

dict, dict