Metadata and attributes#

Note at the moment this tutorial is only applicable to surface data and not to the other data types.

0. Using the tutorial object store#

To avoid adding the example data we use in this tutorial to your normal object store, we need to tell OpenGHG to use a separate sandboxed object store that we’ll call the tutorial store. To do this we use the use_tutorial_store function from openghg.tutorial. This sets the OPENGHG_TUT_STORE environment variable for this session and won’t affect your use of OpenGHG outside of this tutorial.

from openghg.tutorial import use_tutorial_store

use_tutorial_store()

1. Add example observation data#

For this tutorial, we can add some example Tacolnestion data to the tutorial object store.

from openghg.tutorial import retrieve_example_data

data_url = "https://github.com/openghg/example_data/raw/main/timeseries/tac_example.tar.gz"

tac_data = retrieve_example_data(url=data_url)

from openghg.standardise import standardise_surface

standardise_surface(filepaths=tac_data, source_format="CRDS", site="TAC", network="DECC")

We’ll also retrieve some dummy alternative data which we can use to demonstrate how to deal with mismatching metadata and attributes.

from openghg.tutorial import retrieve_example_data

data_url = "https://github.com/openghg/example_data/raw/main/timeseries/tac_dummy_attr_mismatch_example.tar.gz"

dummy_data = retrieve_example_data(url=data_url)

We will demonstrate how to add this to the object store below.

2. Metadata and attributes#

Within OpenGHG, metadata is used as a way to categorise a data source by applying a unique set of keys. This is used to distinguish between stored data and allows this to be searchable.

For CF-compliance and ease of use, we also provide a set of internal attributes stored alongside the data and within the netcdf file (as Dataset.attrs when using xarray for instance).

The metadata and attributes stored do not necessarily need to be the same for a given data source. Indeed, we may want different information stored as tags in the metadata and more extensive details included in the attributes. However, for any overlapping tags (keys) between the metadata and the attributes these values must match.

For the current observation data we have available we can retrieve the carbon dioxide data for the Tacolneston Site at the 185m inlet.

from openghg.retrieve import get_obs_surface

co2_data = get_obs_surface(site="tac", species="co2", inlet="185m")

Previously, we have focussed on the data attribute to access the stored data for this output. We can access the metadata details in a similiar way by looking at the metadata attribute:

co2_data.metadata

Output:

{'data_type': 'surface',
 'site': 'tac',
 'instrument': 'picarro',
 'sampling_period': '3600.0',
 'inlet': '185m',
 'port': '10',
 'type': 'air',
 'network': 'decc',
 'species': 'co2',
 'calibration_scale': 'wmo-x2007',
 'long_name': 'tacolneston',
 'inlet_height_magl': '185m',
 'data_owner': "Simon O'Doherty",
 'data_owner_email': 's.odoherty@bristol.ac.uk',
 'station_longitude': 1.13872,
 'station_latitude': 52.51775,
 'station_long_name': 'Tacolneston Tower, UK',
 'station_height_masl': 50.0,
 'uuid': 'f3e1ef46-8907-4096-8215-19bd6e1c55e3',
 'comment': 'Cavity ring-down measurements. Output from GCWerks',
 'conditions_of_use': 'Ensure that you contact the data owner at the outset of your project.',
 'source': 'In situ measurements of air',
 'Conventions': 'CF-1.8',
 'file_created': '2022-12-13 10:23:34.956121+00:00',
 'processed_by': 'OpenGHG_Cloud',
 'sampling_period_unit': 's',
 'scale': 'WMO-X2007'}

You will see this is stored as a dictionary containing the unique keys associated with this data. These details are is what allows openghg to search and retrieve specific data sources.

The attributes are associated internally with the data itself:

co2_ds = co2_data.data
co2_ds

Output:

xarray.Dataset
    Dimensions:
    time: 39114
    Coordinates:
    time
    (time)
    datetime64[ns]
    2013-01-31T00:13:28 ... 2017-12-...
    Data variables:
        mf             (time) float64 401.6 403.4 403.1 ... 411.1 411.1
        mf_variability (time) float64 0.155 0.088 0.204 ... 0.421 0.325
        mf_number_...  (time) float64 259.0 251.0 252.0 ... 596.0 596.0
    Indexes: (1)
    Attributes: (25)

To access the “Attributes” we can use the attrs keyword for xarray Datasets.

co2_ds.attrs

Output:

{'data_type': 'surface',
'site': 'tac',
'instrument': 'picarro',
'sampling_period': '3600.0',
'inlet': '185m',
'port': '10',
'type': 'air',
'network': 'decc',
'species': 'co2',
'calibration_scale': 'wmo-x2007',
'long_name': 'tacolneston',
'inlet_height_magl': '185m',
'data_owner': "Simon O'Doherty",
'data_owner_email': 's.odoherty@bristol.ac.uk',
'station_longitude': 1.13872,
'station_latitude': 52.51775,
'station_long_name': 'Tacolneston Tower, UK',
'station_height_masl': 50.0,
'uuid': 'f3e1ef46-8907-4096-8215-19bd6e1c55e3',
'comment': 'Cavity ring-down measurements. Output from GCWerks',
'conditions_of_use': 'Ensure that you contact the data owner at the outset of your project.',
'source': 'In situ measurements of air',
'Conventions': 'CF-1.8',
'file_created': '2022-12-13 10:23:34.956121+00:00',
'processed_by': 'OpenGHG_Cloud',
'sampling_period_unit': 's',
'scale': 'WMO-X2007'}

Storing attributes in this way means it’s easy to create a CF-compliant netcdf file from the standardised data in the object store, for example using the to_netcdf() method on our Dataset:

# co2_ds.to_netcdf(...)

Here we would uncomment this and substitue for a filepath.

3. Resolving mismatches#

When the metadata and attributes are created as part of the openghg standardisation process (and when using retrieve_atmospheric), these sets of details are often collated from different sources.

In general:

attributes are drawn from internal attributes from the data
metadata is drawn from additional external details including user inputs and the openghg/openghg_defs

data repository.

Depending on the standardisation procedure, there are cases where there may be a mismatch between these two sets of details. For instance, you may wish to specify a station long name when adding a new data file as an input but this conflicts with attributes stored within the data file itself. You may also find when retrieving data from an external source, such as the ICOS Carbon Portal, the attributes stored alongside retrieved data do not match to our definitions stored within the openghg/openghg_defs site_info details for that site.

Though overlapping details stored in the attributes and metadata must match, how these mismatches are handled is up to the user. When adding new data either via standardise_surface (or pulling data using retrieve_atmopsheric) this can be done through the update_mismatch keyword.

In Step 1, you should have already retrieved some dummy data we can use to demonstrate this. This will have been created as a variable called dummy_data which we will use below. Check this has been run if you’re unable to access this variable.

standardise_surface(filepaths=dummy_data,
                    source_format="openghg",
                    site="TAC",
                    network="DECC",
                    inlet="998m",
                    instrument="picarro",
                    sampling_period="1H")

Output:

---------------------------------------------------------------------------
AttrMismatchError                         Traceback (most recent call last)

...

AttrMismatchError: Metadata mismatch / value not within tolerance for the following keys:
- 'station_long_name', metadata: Tacolneston Tower, UK, attributes: ATTRIBUTE DATA
- 'station_height_masl', metadata: 64, attributes: 50.0
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

If we try to add this dummy data, we’ll see that this fails with a AttrMismatchError. This is because some details stored within the input file (the attributes) don’t match to the details within our stored data within the openghg/openghg_defs [site_info details](openghg/openghg_defs) for that site. By default update_mismatch is set to “never” which means this will produce an error rather than guessing how to resolve this.

The error message above also tells us what doesn’t match:

station_long_name
- metadata: Tacolneston Tower, UK
- attributes: ATTRIBUTE DATA
station_height_masl
- metadata: 64
- attributes: 50.0

We can choose how we want to resolve this using the options for the update_mismatch keyword:

“from_source” (or “attributes”) - use the value(s) included within the current attributes
“from_definition” (or “metadata”) - use the value(s) included within the current metadata

In this case, we choose to use the details from the metadata (derived from site_info details) by running standardise_surface again but this time using update_mismatch=”from_definition”.

standardise_surface(filepaths=dummy_data,
                    source_format="openghg",
                    site="TAC",
                    species="co2",
                    network="DECC",
                    inlet="998m",
                    instrument="picarro",
                    sampling_period="1H",
                    update_mismatch="from_definition")

This should now run without error (warnings will be printed and logged instead).

dummy_data = get_obs_surface(site="tac", species="co2", inlet="998m")

We can look at the station_long_name stored within the metadata:

dummy_data.metadata["station_long_name"]

Output:

'Tacolneston Tower, UK'

and attributes:

dummy_data.data.attrs["station_long_name"]

Output:

'Tacolneston Tower, UK'

to check this is what we expected.

4. Cleanup#

If you’re finished with the data in this tutorial you can cleanup the tutorial object store using the clear_tutorial_store function.

from openghg.tutorial import clear_tutorial_store

clear_tutorial_store()