Retrieve#

These handle the retrieval of data from the object store.

Search functions#

We have a number of search functions, most customised to the data type, which we hope will make it easier for users to find the data they require from the object store.

Surface observations#

To search for surface observations we recommend the use of search_surface.

openghg.retrieve.search_surface(species=None, site=None, inlet=None, height=None, instrument=None, measurement_type=None, source_format=None, network=None, start_date=None, end_date=None, data_source=None, sampling_height=None, icos_data_level=None)[source]#

Cloud object store search

Parameters:
  • species (Union[str, List[str], None]) – Species

  • site (Union[str, List[str], None]) – Three letter site code

  • inlet (Union[str, List[str], None]) – Inlet height above ground level in metres

  • height (Union[str, List[str], None]) – Alias for inlet

  • instrument (Union[str, List[str], None]) – Instrument name

  • measurement_type (Union[str, List[str], None]) – Measurement type

  • data_type – Data type e.g. “surface”, “column”, “emissions” See openghg.store.spec.define_data_types() for full details.

  • start_date (Union[str, List[str], None]) – Start date

  • end_date (Union[str, List[str], None]) – End date

  • data_source (Optional[str]) – Source of data, e.g. noaa_obspack, icoscp, ceda_archive. This

  • sources. (argument only needs to be used to narrow the search to data solely from these) –

  • sampling_height (Optional[str]) – Sampling height of measurements

  • icos_data_level (Optional[int]) – ICOS data level, see ICOS documentation

Returns:

SearchResults object

Return type:

SearchResults

Flux data#

openghg.retrieve.search_flux(species=None, source=None, domain=None, start_date=None, end_date=None, high_time_resolution=None, period=None, continuous=None)[source]#

Search for emissions data

Parameters:
  • species (Optional[str]) – Species name

  • domain (Optional[str]) – Emissions domain

  • source (Optional[str]) – Emissions source

  • source_format – Type of data being input e.g. openghg (internal format)

  • high_time_resolution (Optional[bool]) – If this is a high resolution file

  • period (Union[str, tuple, None]) –

    Period of measurements. Only needed if this can not be inferred from the time coords If specified, should be one of:

    • ”yearly”, “monthly”

    • suitable pandas Offset Alias

    • tuple of (value, unit) as would be passed to pandas.Timedelta function

  • continuous (Optional[bool]) – Whether time stamps have to be continuous.

Returns:

SearchResults object

Return type:

SearchResults

Boundary conditions data#

openghg.retrieve.search_bc(species=None, bc_input=None, domain=None, start_date=None, end_date=None, period=None, continuous=None)[source]#

Search for boundary condition data

Parameters:
  • species (Optional[str]) – Species name

  • bc_input (Optional[str]) – Input used to create boundary conditions. For example: - a model name such as “MOZART” or “CAMS” - a description such as “UniformAGAGE” (uniform values based on AGAGE average)

  • domain (Optional[str]) – Region for boundary conditions

  • start_date (Optional[str]) – Start date (inclusive) for boundary conditions

  • end_date (Optional[str]) – End date (exclusive) for boundary conditions

  • period (Union[str, tuple, None]) –

    Period of measurements. Only needed if this can not be inferred from the time coords If specified, should be one of:

    • ”yearly”, “monthly”

    • suitable pandas Offset Alias

    • tuple of (value, unit) as would be passed to pandas.Timedelta function

  • continuous (Optional[bool]) – Whether time stamps have to be continuous.

Returns:

SearchResults object

Return type:

SearchResults

Eulerian data#

openghg.retrieve.search_eulerian(model=None, species=None, start_date=None, end_date=None)[source]#

Search for eulerian data

Parameters:
  • model (Optional[str]) – Eulerian model name

  • species (Optional[str]) – Species name

  • start_date (Optional[str]) – Start date (inclusive) associated with model run

  • end_date (Optional[str]) – End date (exclusive) associated with model run

Returns:

SearchResults object

Return type:

SearchResults

Column / satellite data#

openghg.retrieve.search_column(satellite=None, domain=None, selection=None, site=None, species=None, network=None, instrument=None, platform=None)[source]#

Search column data

Parameters:
  • satellite (Optional[str]) – Name of satellite (if relevant)

  • domain (Optional[str]) – For satellite only. If data has been selected on an area include the identifier name for domain covered. This can map to previously defined domains (see openghg_defs “domain_info.json” file) or a newly defined domain.

  • selection (Optional[str]) – For satellite only, identifier for any data selection which has been performed on satellite data. This can be based on any form of filtering, binning etc. but should be unique compared to other selections made e.g. “land”, “glint”, “upperlimit”. If not specified, domain will be used.

  • site (Optional[str]) – Site code/name (if relevant). Can include satellite OR site.

  • species (Optional[str]) – Species name or synonym e.g. “ch4”

  • instrument (Optional[str]) – Instrument name e.g. “TANSO-FTS”

  • network (Optional[str]) – Name of in-situ or satellite network e.g. “TCCON”, “GOSAT”

  • platform (Optional[str]) – Type of platform. Should be one of: - “satellite” - “site”

Returns:

SearchResults object

Return type:

SearchResults

Footprints#

openghg.retrieve.search_footprints(site=None, inlet=None, domain=None, model=None, height=None, metmodel=None, species=None, start_date=None, end_date=None, network=None, period=None, continuous=None, high_spatial_res=None, high_time_res=None, short_lifetime=None)[source]#

Search for footprints data

Parameters:
  • site (Optional[str]) – Site name

  • inlet (Optional[str]) – Height above ground level in metres

  • domain (Optional[str]) – Domain of footprints

  • model (Optional[str]) – Model used to create footprint (e.g. NAME or FLEXPART)

  • height (Optional[str]) – Alias for inlet

  • metmodel (Optional[str]) – Underlying meteorlogical model used (e.g. UKV)

  • species (Optional[str]) – Species name. Only needed if footprint is for a specific species e.g. co2 (and not inert)

  • network (Optional[str]) – Network name

  • period (Union[str, tuple, None]) – Period of measurements. Only needed if this can not be inferred from the time coords

  • continuous (Optional[bool]) – Whether time stamps have to be continuous.

  • retrieve_met – Whether to also download meterological data for this footprints area

  • high_spatial_res (Optional[bool]) – Indicate footprints include both a low and high spatial resolution.

  • high_time_res (Optional[bool]) – Indicate footprints are high time resolution (include H_back dimension) Note this will be set to True automatically if species=”co2” (Carbon Dioxide).

  • short_lifetime (Optional[bool]) – Indicate footprint is for a short-lived species. Needs species input. Note this will be set to True if species has an associated lifetime.

Returns:

SearchResults object

Return type:

SearchResults

General#

For a more general search you can use the search function directly. This function accepts any number of keyword arguments.

openghg.retrieve.search(**kwargs)[source]#

Search for observations data. Any keyword arguments may be passed to the the function and these keywords will be used to search the metadata associated with each Datasource.

This function detects the running environment and routes the call to either the cloud or local search function.

Example / commonly used arguments are given below.

Parameters:
  • species – Terms to search for in Datasources

  • locations – Where to search for the terms in species

  • inlet – Inlet height such as 100m

  • instrument – Instrument name such as picarro

  • find_all – Require all search terms to be satisfied

  • start_date – Start datetime for search.

  • epoch (If None a start datetime of UNIX) –

  • end_date – End datetime for search.

  • set (If None an end datetime of the current datetime is) –

Returns:

SearchResults object is results found, otherwise None

Return type:

SearchResults or None

Retrieving from other data sources#

ICOS#

OpenGHG can retrieve data from the ICOS Carbon Portal.

openghg.retrieve.icos.retrieve_atmospheric(site, species=None, sampling_height=None, start_date=None, end_date=None, force_retrieval=False, data_level=2, update_metadata_mismatch=False)[source]#

Retrieve ICOS atmospheric measurement data. If data is found in the object store it is returned. Otherwise data will be retrieved from the ICOS Carbon Portal. Data retrieval from the Carbon Portal may take a short time. If only a single data source is found an ObsData object is returned, if multiple a list of ObsData objects if returned, if nothing then None.

Parameters:
  • site (str) – Site code

  • species (Union[str, List, None]) – Species name

  • start_date (Optional[str]) – Start date

  • end_date (Optional[str]) – End date

  • force_retrieval (bool) – Force the retrieval of data from the ICOS Carbon Portal

  • data_level (int) – ICOS data level (1, 2)

  • 1 (- Data level) – Near Real Time Data (NRT) or Internal Work data (IW).

  • 2 (- Data level) – The final quality checked ICOS RI data set, published by the CFs, to be distributed through the Carbon Portal. This level is the ICOS-data product and free available for users.

  • https (See) – //icos-carbon-portal.github.io/pylib/modules/#stationdatalevelnone

  • bypass_call – Bypass the remote function call, used to shortcut calls within a the serverless

  • environment. (function call) –

  • update_metadata_mismatch (bool) – If metadata derived from ICOS Header does not match to derived attributes, update metadata to match to attributes. Otherwise a AttrMismatchError will be raised.

Return type:

Union[ObsData, List[ObsData], None]

Returns:

ObsData, list[ObsData] or None

CEDA#

Pulling from the CEDA archive is also possible. After finding the URL to the dataset you require you can retrieve it using

openghg.retrieve.ceda.retrieve_surface(site=None, species=None, inlet=None, url=None, force_retrieval=False, additional_metadata=None)[source]#

Retrieve surface measurements from the CEDA archive. This function will route the call to either local or cloud functions based on the environment.

Parameters:
  • site (Optional[str]) – Site name

  • species (Optional[str]) – Species name

  • inlet (doesn't contain everythging we need. At the moment we try and find site and) – Inlet height

  • url (Optional[str]) – URL of data in CEDA archive

  • force_retrieval (bool) – Force the retrieval of data from a URL

  • additional_metadata (Optional[Dict]) – Additional metadata to pass if the returned data

  • inlet

  • attributes. (keys if they aren't found in the dataset's) –

  • example (For) – {“site”: “AAA”, “inlet”: “10m”}

Returns:

ObsData if data found / retrieved successfully.

Return type:

ObsData or None

Example

To retrieve new data from the CEDA archive using a URL >>> retrieve_surface(url=https://dap.ceda.ac.uk/badc/…) To retrieve already cached data from the object store >>> retrieve_surface(site=”BSD”, species=”ch4)

Specific retrieval functions#

openghg.retrieve.get_obs_surface(site, species, inlet=None, height=None, start_date=None, end_date=None, average=None, network=None, instrument=None, calibration_scale=None, keep_missing=False, skip_ranking=False)[source]#

This is the equivalent of the get_obs function from the ACRG repository.

Usage and return values are the same whilst implementation may differ.

Parameters:
  • site (str) – Site of interest e.g. MHD for the Mace Head site.

  • species (str) – Species identifier e.g. ch4 for methane.

  • start_date (Union[str, Timestamp, None]) – Output start date in a format that Pandas can interpret

  • end_date (Union[str, Timestamp, None]) – Output end date in a format that Pandas can interpret

  • inlet (Optional[str]) – Inlet height above ground level in metres

  • height (Optional[str]) – Alias for inlet

  • average (Optional[str]) – Averaging period for each dataset. Each value should be a string of

  • "2H" (the form e.g.) –

  • "30min" (should match pandas offset aliases format) –

  • keep_missing (bool) – Keep missing data points or drop them.

  • network (Optional[str]) – Network for the site/instrument (must match number of sites).

  • instrument (Optional[str]) – Specific instrument for the sipte (must match number of sites).

  • calibration_scale (Optional[str]) – Convert to this calibration scale

Returns:

ObsData object if data found, else None

Return type:

ObsData or None

openghg.retrieve.get_flux(species, source, domain, start_date=None, end_date=None, time_resolution=None)[source]#

The flux function reads in all flux files for the domain and species as an xarray Dataset. Note that at present ALL flux data is read in per species per domain or by emissions name. To be consistent with the footprints, fluxes should be in mol/m2/s.

Parameters:
  • species (str) – Species name

  • source (str) – Source name

  • domain (str) – Domain e.g. EUROPE

  • start_date (Union[str, Timestamp, None]) – Start date

  • end_date (Union[str, Timestamp, None]) – End date

  • time_resolution (Optional[str]) – One of [“standard”, “high”]

Returns:

FluxData object

Return type:

FluxData

openghg.retrieve.get_footprint(site, domain, inlet=None, height=None, model=None, start_date=None, end_date=None, species=None)[source]#

Get footprints from one site.

Parameters:
  • site (str) – The name of the site given in the footprints. This often matches to the site name but if the same site footprints are run with a different met and they are named slightly differently from the obs file. E.g. site=”DJI”, site_modifier = “DJI-SAM” - station called DJI, footprints site called DJI-SAM

  • domain (str) – Domain name for the footprints

  • inlet (Optional[str]) – Height above ground level in metres

  • height (Optional[str]) – Alias for inlet

  • start_date (Union[str, Timestamp, None]) – Output start date in a format that Pandas can interpret

  • end_date (Union[str, Timestamp, None]) – Output end date in a format that Pandas can interpret

  • species (Optional[str]) – Species identifier e.g. “co2” for carbon dioxide. Only needed if species needs a modified footprints from the typical 30-day footprints appropriate for a long-lived species (like methane) e.g. for high time resolution (co2) or is a short-lived species.

Returns:

FootprintData dataclass

Return type:

FootprintData

openghg.retrieve.get_bc(species, domain, bc_input=None, start_date=None, end_date=None)[source]#

Get boundary conditions for a given species, domain and bc_input name.

Parameters:
  • species (str) – Species name

  • bc_input (Optional[str]) – Input used to create boundary conditions. For example: - a model name such as “MOZART” or “CAMS” - a description such as “UniformAGAGE” (uniform values based on AGAGE average)

  • domain (str) – Region for boundary conditions e.g. EUROPE

  • start_date (Union[str, Timestamp, None]) – Start date

  • end_date (Union[str, Timestamp, None]) – End date

Returns:

BoundaryConditionsData object

Return type:

BoundaryConditionsData