Standardise - data#

Each of these functions parses a specific type of data file and returns a dictionary containing the data and metadata.

Surface observations#

openghg.standardise.surface.parse_aqmesh(data_filepath, metadata_filepath, sampling_period=None, **kwargs)[source]#

Read AQMesh data files

Parameters:
  • data_filepath (Union[str, Path]) – Data filepath

  • metadata_filepath (Union[str, Path]) – Metadata filepath

  • sampling_period (Optional[str]) – Measurement sampling period (str)

Returns:

Dictionary of data

Return type:

dict

openghg.standardise.surface.parse_beaco2n(data_filepath, site, network, inlet, instrument='shinyei', sampling_period=None, **kwargs)[source]#

Read BEACO2N data files

Parameters:
  • data_filepath (Union[str, Path]) – Data filepath

  • site (str) – Site name

  • network (str) – Network name

  • inlet (str) – Inlet height in metres

  • instrument (Optional[str]) – Instrument name

  • sampling_period (Optional[str]) – Measurement sampling period

Returns:

Dictionary of data

Return type:

dict

openghg.standardise.surface.parse_btt(data_filepath, site='BTT', network='LGHG', inlet=None, instrument=None)[source]#

Reads NPL data files and returns the UUIDS of the Datasources the processed data has been assigned to

Parameters:
  • data_filepath (Union[str, Path]) – Path of file to load

  • site (Optional[str]) – Site name

Returns:

Dictionary of gas data

Return type:

dict

openghg.standardise.surface.parse_cranfield(data_filepath, site=None, network=None, inlet=None, instrument=None, sampling_period=None, measurement_type=None, **kwargs)[source]#

Creates a CRDS object holding data stored within Datasources

Parameters:
  • filepath – Path of file to load

  • data_filepath (Union[str, Path]) – Filepath of data to be read

  • site (Optional[str]) – Name of site

  • network (Optional[str]) – Name of network

Returns:

Dictionary of gas data

Return type:

dict

openghg.standardise.surface.parse_crds(data_filepath, site, network, inlet=None, instrument=None, sampling_period=None, measurement_type=None, drop_duplicates=True, **kwargs)[source]#

Parses a CRDS data file and creates a dictionary of xarray Datasets ready for storage in the object store.

Parameters:
  • data_filepath (Union[str, Path]) – Path to file

  • site (str) – Three letter site code

  • network (str) – Network name

  • inlet (Optional[str]) – Inlet height

  • instrument (Optional[str]) – Instrument name

  • sampling_period (Union[str, float, int, None]) – Sampling period in seconds

  • measurement_type (Optional[str]) – Measurement type e.g. insitu, flask

  • drop_duplicates (bool) – Drop measurements at duplicate timestamps, keeping the first.

Returns:

Dictionary of gas data

Return type:

dict

openghg.standardise.surface.parse_eurocom(data_filepath, site, sampling_period, network=None, inlet=None, instrument=None)[source]#

Parses EUROCOM data files into a format expected by OpenGHG

Parameters:
  • data_filepath (Union[str, Path]) – Path of file to read

  • site (str) – Site code

  • sampling_period (str) – Sampling period in seconds

  • network (Optional[str]) – Network name

  • Inlet – Inlet height in metres

  • Instrument – Instrument name

Returns:

Dictionary of measurement data

Return type:

dict

openghg.standardise.surface.parse_gcwerks(data_filepath, precision_filepath, site, network, inlet=None, instrument=None, sampling_period=None, measurement_type=None)[source]#

Reads a GC data file by creating a GC object and associated datasources

Parameters:
  • data_filepath (Union[str, Path]) – Path of data file

  • precision_filepath (Union[str, Path]) – Path of precision file

  • site (str) – Three letter code or name for site

  • instrument (Optional[str]) – Instrument name

  • network (str) – Network name

Returns:

Dictionary of source_name : UUIDs

Return type:

dict

openghg.standardise.surface.parse_noaa(data_filepath, site, measurement_type, inlet=None, network='NOAA', instrument=None, sampling_period=None, **kwarg)[source]#

Read NOAA data from raw text file or ObsPack NetCDF

Parameters:
  • data_filepath (Union[str, Path]) – Data filepath

  • site (str) – Three letter site code

  • inlet (Optional[str]) – Inlet height (as value unit e.g. “10m”)

  • measurement_type (str) – One of (“flask”, “insitu”, “pfp”)

  • network (str) – Network, defaults to NOAA

  • instrument (Optional[str]) – Instrument name

  • sampling_period (Optional[str]) – Sampling period

Returns:

Dictionary of data and metadata

Return type:

dict

openghg.standardise.surface.parse_npl(data_filepath, site='NPL', network='LGHG', inlet=None, instrument=None, sampling_period=None, measurement_type=None)[source]#

Reads NPL data files and returns the UUIDS of the Datasources the processed data has been assigned to

Parameters:
  • data_filepath (Union[str, Path]) – Path of file to load

  • site (str) – Site name

  • network (str) – Network, defaults to LGHG

  • inlet (Optional[str]) – Inlet height. Will be inferred if not specified

  • instrument (Optional[str]) – Instrument name

  • sampling_period (Optional[str]) – Sampling period

  • measurement_type (Optional[str]) – Type of measurement taken e.g.”flask”, “insitu”

Returns:

UUIDs of Datasources data has been assigned to

Return type:

list

openghg.standardise.surface.parse_tmb(data_filepath, site='TMB', network='LGHG', inlet=None, instrument=None, sampling_period=None, measurement_type=None, **kwargs)[source]#

Reads THAMESBARRIER data files and returns the UUIDS of the Datasources the processed data has been assigned to

Parameters:
  • data_filepath (Union[str, Path]) – Path of file to load

  • site (str) – Site name

  • network (str) – Network, defaults to LGHG

  • inlet (Optional[str]) – Inlet height. Will be inferred if not specified

  • instrument (Optional[str]) – Instrument name

  • sampling_period (Optional[str]) – Sampling period

  • measurement_type (Optional[str]) – Type of measurement taken e.g.”flask”, “insitu”

Returns:

UUIDs of Datasources data has been assigned to

Return type:

list

Column data#

openghg.standardise.column.parse_openghg(data_filepath, satellite=None, domain=None, selection=None, site=None, species=None, network=None, instrument=None, platform='satellite', **kwargs)[source]#

Parse and extract data from pre-formatted netcdf file which already matches expected OpenGHG format.

The arguments specified below are the metadata needed to store this surface observation file within the object store. If these keywords are not included within the attributes of the netcdf file being passed then these arguments must be specified.

For column data this can either be a satellite (e.g. satellite=”GOSAT”) or a site (site=”RUN”, network=”TCCON”). Either can be specified or this function will attempt to extract this from the data file.

Parameters:
  • data_filepath (Union[str, Path]) – Path of observation file

  • satellite (Optional[str]) – Name of satellite (if relevant)

  • domain (Optional[str]) – For satellite only. If data has been selected on an area include the identifier name for domain covered. This can map to previously defined domains (see openghg_defs “domain_info.json” file) or a newly defined domain.

  • selection (Optional[str]) – For satellite only, identifier for any data selection which has been performed on satellite data. This can be based on any form of filtering, binning etc. but should be unique compared to other selections made e.g. “land”, “glint”, “upperlimit”. If not specified, domain will be used.

  • site (Optional[str]) – Site code/name (if relevant). Can include satellite OR site.

  • species (Optional[str]) – Species name or synonym e.g. “ch4”

  • instrument (Optional[str]) – Instrument name e.g. “TANSO-FTS”

  • network (Optional[str]) – Name of in-situ or satellite network e.g. “TCCON”, “GOSAT”

  • platform (str) – Type of platform. Should be one of: - “satellite” - “site” Note: this will be superceded if site or satellite keywords are specified.

  • kwargs (str) – Any additional attributes to be associated with the data.

Returns:

Dictionary of source_name : data, metadata, attributes

Return type:

Dict

Emissions / flux#

openghg.standardise.column.parse_openghg(data_filepath, satellite=None, domain=None, selection=None, site=None, species=None, network=None, instrument=None, platform='satellite', **kwargs)[source]#

Parse and extract data from pre-formatted netcdf file which already matches expected OpenGHG format.

The arguments specified below are the metadata needed to store this surface observation file within the object store. If these keywords are not included within the attributes of the netcdf file being passed then these arguments must be specified.

For column data this can either be a satellite (e.g. satellite=”GOSAT”) or a site (site=”RUN”, network=”TCCON”). Either can be specified or this function will attempt to extract this from the data file.

Parameters:
  • data_filepath (Union[str, Path]) – Path of observation file

  • satellite (Optional[str]) – Name of satellite (if relevant)

  • domain (Optional[str]) – For satellite only. If data has been selected on an area include the identifier name for domain covered. This can map to previously defined domains (see openghg_defs “domain_info.json” file) or a newly defined domain.

  • selection (Optional[str]) – For satellite only, identifier for any data selection which has been performed on satellite data. This can be based on any form of filtering, binning etc. but should be unique compared to other selections made e.g. “land”, “glint”, “upperlimit”. If not specified, domain will be used.

  • site (Optional[str]) – Site code/name (if relevant). Can include satellite OR site.

  • species (Optional[str]) – Species name or synonym e.g. “ch4”

  • instrument (Optional[str]) – Instrument name e.g. “TANSO-FTS”

  • network (Optional[str]) – Name of in-situ or satellite network e.g. “TCCON”, “GOSAT”

  • platform (str) – Type of platform. Should be one of: - “satellite” - “site” Note: this will be superceded if site or satellite keywords are specified.

  • kwargs (str) – Any additional attributes to be associated with the data.

Returns:

Dictionary of source_name : data, metadata, attributes

Return type:

Dict

Metadata#

These ensure the metadata and attributes stored with data are correct.

openghg.standardise.meta.assign_attributes(data, site=None, network=None, sampling_period=None, update_metadata_mismatch=False, site_filename=None, species_filename=None)[source]#

Assign attributes to each site and species dataset. This ensures that the xarray Datasets produced are CF 1.7 compliant. Some of the attributes written to the Dataset are saved as metadata to the Datasource allowing more detailed searching of data.

If accessing underlying stored site or species definitions, this will be accessed from the openghg/supplementary_data repository by default.

Parameters:
  • data (Dict) – Dictionary containing data, metadata and attributes

  • site (Optional[str]) – Site code

  • sampling_period (Union[str, float, int, None]) – Number of seconds for which air sample is taken. Only for time variable attribute

  • network (Optional[str]) – Network name

  • update_metadata_mismatch (bool) – If current metadata does not match to attributes update metadata.

  • site_filename (Union[str, Path, None]) – Alternative site info file

  • species_filename (Union[str, Path, None]) – Alternative species info file

Returns:

Dictionary of combined data with correct attributes assigned to Datasets

Return type:

dict

openghg.standardise.meta.get_attributes(ds, species, site, network=None, global_attributes=None, units=None, scale=None, sampling_period=None, date_range=None, site_filename=None, species_filename=None)[source]#

This function writes attributes to an xarray.Dataset so that they conform with the CF Convention v1.6

Attributes of the xarray DataSet are modified, and variable names are changed

If accessing underlying stored site or species definitions, this will be accessed from the openghg/supplementary_data repository by default.

Variable naming related to species name will be defined using define_species_label() function.

Parameters:
  • ds (Dataset) – Should contain variables such as “ch4”, “ch4 repeatability”. Must have a “time” dimension.

  • species (str) – Species name. e.g. “CH4”, “HFC-134a”, “dCH4C13”

  • site (str) – Three-letter site code

  • network (Optional[str]) – Network site is associated with

  • global_attribuates – Dictionary containing any info you want to add to the file header (e.g. {“Contact”: “Contact_Name”})

  • units (Optional[str]) – This routine will try to guess the units unless this is specified. Options are in units_interpret

  • scale (Optional[str]) – Calibration scale for species.

  • sampling_period (Union[str, float, int, None]) – Number of seconds for which air sample is taken. Only for time variable attribute

  • date_range (Optional[List[str]]) – Start and end date for output If you only want an end date, just put a very early start date (e.g. [“1900-01-01”, “2010-01-01”])

  • site_filename (Union[str, Path, None]) – Alternative site info file

  • species_filename (Union[str, Path, None]) – Alternative species info file

Return type:

Dataset

openghg.standardise.meta.assign_flux_attributes(data, species=None, source=None, domain=None, units='mol/m2/s', prior_info_dict=None)[source]#

Assign attributes for the input flux dataset within dictionary based on metadata and passed arguments.

Parameters:
  • data (Dict) – Dictionary containing data, metadata and attributes

  • species (Optional[str]) – Species name

  • source (Optional[str]) – Source name

  • domain (Optional[str]) – Domain name

  • units (str) – Unit values for the “flux” variable. Default = “mol/m2/s”

  • prior_info_dict (Optional[dict]) –

    Dictionary of additional ‘prior’ information about for the emissions sources. Expect this to be of the form e.g.

    {“EDGAR”: {“version”: “v4.3.2”,

    ”raw_resolution”: “0.1 degree x 0.1 degree”, “reference”: “http://edgar.jrc.ec.europa.eu/overview.php?v=432_GHG” …},

    …}

Returns:

Same format as inputted but with updated “data” component (Dataset)

Return type:

Dict

openghg.standardise.meta.define_species_label(species, species_filename=None)[source]#

Define standardised label to use for observation datasets. This uses the data stored within openghg_defs/data/site_info JSON file by default with alternative names (‘alt’) defined within.

Formatting:
  • species label will be all lower case

  • any spaces will be replaced with underscores

  • if species or synonym cannot be found, species name will used

    but with any hyphens taken out (see also openghg.util.clean_string function)

Note: Suggested naming for isotopologues should be d<species><isotope>, e.g. dCH4C13, or dCO2C14

Parameters:
  • species (str) – Species name.

  • species_filename (Union[str, Path, None]) – Alternative species info file.

Returns:

Both the species label to be used exactly and the original attribute

key needed to extract additional data from the ‘site_info.json’ attributes file.

Return type:

str, str

Example

>>> define_species_label("methane")
    ("ch4", "CH4")
>>> define_species_label("radon")
    ("rn", "Rn")
>>> define_species_label("cfc-11")
    ("cfc11", "CFC11")
>>> define_species_label("CH4C13")
    ("dch4c13", "DCH4C13")
openghg.standardise.meta.metadata_default_keys()[source]#

Define default values expected within ObsSurface metadata

Return type:

List

openghg.standardise.meta.sync_surface_metadata(metadata, attributes, keys_to_add=None, update_mismatch=False)[source]#

Makes sure any duplicated keys between the metadata and attributes dictionaries match and that certain keys are present in the metadata.

Parameters:
  • metadata (Dict) – Dictionary of metadata

  • attributes (Dict) – Attributes

  • keys_to_add (Optional[List]) – Add these keys to the metadata, if not present, based on

  • Note (the attribute values.) – this skips any keys which can’t be

  • values. (copied from the attribute) –

  • update_mismatch (bool) – If case insensitive mismatch is found between an attribute and a metadata value, update the metadata to contain the attribute value. By default this will raise an AttrMismatchError.

Returns:

Copy of metadata updated with attributes

Return type:

dict