Adding observation data#

This tutorial demonstrates how OpenGHG can be used to process new measurement data, search the data present and to retrieve this for analysis and visualisation.

What is an object store?#

Each object and piece of data in the object store is stored at a specific key, which can be thought of as the address of the data. The data is stored in a bucket which in the cloud is a section of the OpenGHG object store. Locally a bucket is just a normal directory in the user’s filesystem specified by the path given in the configuration file at ~/.config/openghg/openghg.conf.

0. Using the tutorial object store#

An object store is a folder with a fixed structure within which openghg can read and write data. To avoid adding the example data we use in this tutorial to your normal object store, we need to tell OpenGHG to use a separate sandboxed object store that we’ll call the tutorial store. To do this we use the use_tutorial_store function from openghg.tutorial. This sets the OPENGHG_TUT_STORE environment variable for this session and won’t affect your use of OpenGHG outside of this tutorial.

from openghg.tutorial import use_tutorial_store

use_tutorial_store()

1. Adding and standardising data#

Note

Outside of this tutorial, if you have write access to multiple object stores you will need to pass the name of the object store you wish to write to to the store argument of the standardise functions.

Source formats#

OpenGHG can process and store several source formats in the object store, including data from the AGAGE, DECC, NOAA, LondonGHG, BEAC2ON networks. The process of adding data to the object store is called standardisation.

To standardise a new data file, you must specify the source format and other keywords for the data. Which keywords need to be specified may be dependent on the source format itself as some details can be inferred from the data or may not be relevant. For the full list of accepted observation inputs and source formats, call the function summary_source_formats:

from openghg.standardise import summary_source_formats

summary = summary_source_formats()

## UNCOMMENT THIS CODE TO SHOW ALL ENTRIES
# import pandas as pd; pd.set_option('display.max_rows', None)

summary
Source format Site code Instrument Network Species file_format Long name Platform
0 CRDS RPB CRDS AGAGE NaN NaN Ragged Point, Barbados surface site
1 CRDS HFD CRDS DECC NaN NaN Heathfield, UK surface site
2 CRDS BSD CRDS DECC NaN NaN Bilsdale, UK surface site
3 CRDS TTA CRDS DECC NaN NaN Angus Tower, UK surface site
4 CRDS RGL CRDS DECC NaN NaN Ridge Hill, UK surface site
... ... ... ... ... ... ... ... ...
339 NOAA WSD NaN NaN NaN NaN Wessington, South Dakota surface site
340 NOAA YON NaN NaN NaN NaN Yonagunijima, Japan surface site
341 NOAA ZEP NaN NaN NaN NaN Zeppelin, Ny Alesund, Norway surface site
342 NPL NPL NaN LGHG NaN NaN National Physical Laboratory surface site
343 NIWA ARH NaN NaN NaN NaN Arrival Heights, New Zealand surface site

344 rows × 8 columns

There may be multiple source formats for a given site. For instance, the Tacolneston site in the UK (site code “TAC”) has four entries:

summary[summary["Site code"] == "TAC"]
Source format Site code Instrument Network Species file_format Long name Platform
5 CRDS TAC CRDS DECC NaN NaN Tacolneston Tower, UK surface site
32 GCWERKS TAC GCMD DECC NaN NaN Tacolneston Tower, UK surface site
34 GCWERKS TAC medusa DECC NaN NaN Tacolneston Tower, UK surface site
302 NOAA TAC NaN NaN NaN NaN Tacolneston Tower, UK surface site

Let’s see what data is available for a given source. First, we’ll list all source formats.

summary["Source format"].unique()

array(['CRDS', 'GCWERKS', 'AGAGE', 'ICOS', 'NOAA', 'NPL', 'NIWA'],
      dtype=object)

Now we’ll find all data with source format "CRDS".

summary[summary["Source format"] == "CRDS"]
Source format Site code Instrument Network Species file_format Long name Platform
0 CRDS RPB CRDS AGAGE NaN NaN Ragged Point, Barbados surface site
1 CRDS HFD CRDS DECC NaN NaN Heathfield, UK surface site
2 CRDS BSD CRDS DECC NaN NaN Bilsdale, UK surface site
3 CRDS TTA CRDS DECC NaN NaN Angus Tower, UK surface site
4 CRDS RGL CRDS DECC NaN NaN Ridge Hill, UK surface site
5 CRDS TAC CRDS DECC NaN NaN Tacolneston Tower, UK surface site

DECC network#

We will start by adding data to the object store from Tacolneston, which is a surface site in the DECC network. (Data at surface sites is measured in-situ.)

First we retrieve the raw data.

from openghg.tutorial import retrieve_example_data

data_url = "https://github.com/openghg/example_data/raw/main/timeseries/tac_example.tar.gz"

tac_data = retrieve_example_data(url=data_url)







Now we add this data to the object store using standardise_surface, passing the
following arguments:


    

  • filepath: list of paths to .dat files
    • 

  • site:  "TAC", the site code for Tacolneston
    • 

  • network: "DECC"
    • 

  • source_format: "CRDS", the type of data we want to process
    • 







from openghg.standardise import standardise_surface

decc_results = standardise_surface(filepath=tac_data, source_format="CRDS", site="TAC", network="DECC")

decc_results









[{'uuid': 'bc0e5994-cd63-42fa-a05b-3f76eff9aa5b',
  'new': True,
  'site': 'tac',
  'instrument': 'picarro',
  'sampling_period': '3600.0',
  'inlet': '54m',
  'network': 'decc',
  'species': 'ch4',
  'source_format': 'CRDS',
  'data_source': 'internal',
  'file': 'tac.picarro.hourly.54m.dat'},
 {'uuid': 'da7847fc-bea9-4488-b677-bd2f0bfaac29',
  'new': True,
  'site': 'tac',
  'instrument': 'picarro',
  'sampling_period': '3600.0',
  'inlet': '54m',
  'network': 'decc',
  'species': 'co2',
  'source_format': 'CRDS',
  'data_source': 'internal',
  'file': 'tac.picarro.hourly.54m.dat'},
 {'uuid': '16032361-4c58-4261-93a8-5092f646bb9e',
  'new': True,
  'site': 'tac',
  'instrument': 'picarro',
  'sampling_period': '3600.0',
  'inlet': '100m',
  'network': 'decc',
  'species': 'ch4',
  'source_format': 'CRDS',
  'data_source': 'internal',
  'file': 'tac.picarro.hourly.100m.dat'},
 {'uuid': '29424a2c-04eb-46bd-acf3-ea919983f926',
  'new': True,
  'site': 'tac',
  'instrument': 'picarro',
  'sampling_period': '3600.0',
  'inlet': '100m',
  'network': 'decc',
  'species': 'co2',
  'source_format': 'CRDS',
  'data_source': 'internal',
  'file': 'tac.picarro.hourly.100m.dat'},
 {'uuid': 'd54d3ed2-c3f8-4dc7-a0e0-96ab9c9b348d',
  'new': True,
  'site': 'tac',
  'instrument': 'picarro',
  'sampling_period': '3600.0',
  'inlet': '185m',
  'network': 'decc',
  'species': 'ch4',
  'source_format': 'CRDS',
  'data_source': 'internal',
  'file': 'tac.picarro.hourly.185m.dat'},
 {'uuid': '5aeed2b4-995c-4fd6-b45b-87afee41fdf6',
  'new': True,
  'site': 'tac',
  'instrument': 'picarro',
  'sampling_period': '3600.0',
  'inlet': '185m',
  'network': 'decc',
  'species': 'co2',
  'source_format': 'CRDS',
  'data_source': 'internal',
  'file': 'tac.picarro.hourly.185m.dat'}]









This extracts the data and metadata from the files,
standardises them, and adds them to our object store. The keywords of site and network,
along with details extracted from the data itself allow us to uniquely store the data.


The returned decc_results dictionary shows how the data
has been stored: each file has been split into several entries, each with a unique ID (UUID).
Each entry is known as a Datasource (see Note on Datasources).


The decc_results output includes details of the processed data and tells
us that the data has been stored correctly. This will also tell us if
any errors have been encountered when trying to access and standardise
this data.






AGAGE data#


OpenGHG can also process data from the AGAGE network.


Historically, the AGAGE network produces output files from GCWERKS alongside a seperate precisions file. If you wish
to use this form of input file, we create a tuple with the data filename and the precisions filename. For example:


First we retrieve example data from the  Cape Grim station in Australia (site code “CGO””).






cgo_url = "https://github.com/openghg/example_data/raw/main/timeseries/capegrim_example.tar.gz"

capegrim_data = retrieve_example_data(url=cgo_url)















capegrim_data is a list of two file paths, one for the data file and one for the precisions file:


[PosixPath('/Users/bm13805/openghg_store/tutorial_store/extracted_files/capegrim.18.C'),
PosixPath('/Users/bm13805/openghg_store/tutorial_store/extracted_files/capegrim.18.precisions.C')]





We put the data file and precisions file into a tuple:






capegrim_tuple = (capegrim_data[0], capegrim_data[1])













We can add these files to the object store in the same way as the DECC
data by including the right arguments:


    

  • filepath: tuple (or list of tuples) with paths to data and precision files
    • 

  • site (site code): "CGO"
    • 

  • network: "AGAGE"
    • 

  • instrument: "medusa"
    • 

  • source_format (data type): "GCWERKS"
    • 







agage_results = standardise_surface(filepath=capegrim_tuple, source_format="GCWERKS", site="CGO",
                              network="AGAGE", instrument="medusa")
agage_results









[{'uuid': 'b7a4fd17-f49a-42b8-94f3-224e90beb951',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'ch4',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': '114c1ef9-49af-4290-975d-042ec52db4aa',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'cfc12',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': '34df8481-339c-40d5-a1a9-ede2223c927b',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'n2o',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': '48a0e155-3099-479f-9c3b-f4c6078ef479',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'cfc11',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': '81174088-e52c-4ff7-84ff-e10818ae9730',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'cfc113',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': '40a7c65f-d8e0-422b-bd44-8de02336eb1a',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'chcl3',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': '362ecbb7-248e-4b2d-a3f1-3d92327c83a9',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'ch3ccl3',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': '9d5db302-afee-4ff3-84b5-046205b90303',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'ccl4',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': '28749937-a65a-4fa4-b732-a7829a8c43d3',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'h2',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': '72750e00-7cb7-49b9-a396-b31a6d45504e',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'co',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'},
 {'uuid': 'bbfd9d33-6474-4144-ac09-28e5c3eb56e1',
  'new': True,
  'instrument': 'medusa',
  'site': 'cgo',
  'network': 'agage',
  'sampling_period': '1200',
  'inlet': '70m',
  'species': 'ne',
  'source_format': 'GCWERKS',
  'data_source': 'internal',
  'file': 'capegrim.18.C'}]









When viewing agage_results there will be a large number of
Datasource UUIDs shown due to the large number of gases in each data
file


However, recently the AGAGE network has begun to also produce netCDF files, which are processed by Matt
Rigby’s agage-archive repository. These files are split by site,
species and instrument and do not need an accompanying precisions file. These can also be read in by the
openghg.standardise.standardise_surface function, with the arguments:


    

  • filepath: filepath to the .nc file
    • 

  • site (site code): "CGO"
    • 

  • source_format (data type): "AGAGE"
    • 

  • network: "AGAGE"
    • 

  • instrument: "medusa"
    • 



The data will be processed in the same way as the old AGAGE data, and stored in the object store accordingly.
Ensure that the source_format argument matches the input filetype, as the two are not compatible.




Note on Datasources#


Datasources are objects that are stored in the object store that hold the data and metadata associated with
each measurement we upload to the platform.


For example, if we upload a file that contains readings for three gas
species from a single site at a specific inlet height OpenGHG will
assign this data to three different Datasources, one for each species.
Metadata such as the site, inlet height, species, network etc are stored
alongside the measurements for easy searching.


Datasources can also handle multiple versions of data from a single
site, so if scales or other factors change multiple versions may be
stored for easy future comparison.








Other keywords#


When adding data in this way there are other keywords which can be used to
distinguish between different data sets as required including:


    

  • instrument: Name of the instrument
    • 

  • sampling_period: The time taken for each measurement to be sampled
    • 

  • data_level: The level of quality control which has been applied to the data.
    • 

  • data_sublevel:  Optional level to include between data levels. Typically for level 1 data where multiple steps of initial QA may have been applied.
    • 

  • dataset_source: Name of the dataset if data is taken from a larger source e.g. from an ObsPack
    • 



See the standardise_surface documentation for a full list of inputs.






Informational keywords#


In addition to the keywords demonstrated for adding data and described above which are used to distinguish
between different data sets being stored, the following informational details can also be added to help describe the data.




Using the tag keyword#


The tag keyword allows one or multiple short labels to be specified which can be the same across multiple
data sources. For instance, data from different sites which is associated with a particular project could all be
added using the same tag. For example below we show how to add the same data as above with a tag:


    

  • Tacolneston (TAC) data with a tag of “project1”
    • 

  • Cape Grim (CGO) data with a tag of both “project1” and “project2”
    • 







from openghg.standardise import standardise_surface

decc_results = standardise_surface(filepath=tac_data,
                                   source_format="CRDS",
                                   site="TAC",
                                   network="DECC",
                                   tag="project1",
                                   force=True)

agage_results = standardise_surface(filepath=capegrim_tuple,
                                    source_format="GCWERKS",
                                    site="CGO",
                                    network="AGAGE",
                                    instrument="medusa",
                                    tag=["project1", "project2"],
                                    force=True)













Note: here we included the force=True keyword as we are adding the same data which has been added in
a previous step of the tutorial - see “Updating existing data” tutorial for more details of this.


As will be covered in the 2. Searching for data section, these keywords can then used when searching the
object store. For the tag keyword this can be used to return all data which includes the chosen tag.






Adding informational keys#


Informational keys and associated values can also be added using the info_metadata input. The most
common example for this would be to add a comment input. For example:


decc_results = standardise_surface(filepath=tac_data,
                                   source_format="CRDS",
                                   site="TAC",
                                   network="DECC",
                                   info_metadata={"comment": "Automatic quality checks have been applied."})





Note that for both info_metadata and tag that these options are available for all data types (not just
observations).








Multiple stores#


If you have write access to more than one object store you’ll need to pass in the name of that store
to the store argument.
So instead of the standardise_surface call above, we’ll tell it to write to our default user object store. This is our default local object store
created when we run openghg --quickstart.


from openghg.standardise import standardise_surface

decc_results = standardise_surface(filepath=tac_data, source_format="CRDS", site="TAC", network="DECC", store="user")





The store argument can be passed to any of the standardise functions in OpenGHG and is required if you have write access
to more than one store.








2. Searching for data#




Searching the object store#


We can search the object store by property using the
search_surface(...) function. This function retrieves all of the metadata associated with the search query from the data in the object store.


For example we can find all sites which have measurements for carbon
tetrafluoride (“cf4”) using the species keyword:






from openghg.retrieve import search_surface

cfc_results = search_surface(species="cfc11")
cfc_results.results













  
    

      
      instrument
      site
      network
      sampling_period
      units
      calibration_scale
      inlet
      species
      data_type
      species_alt
      ...
      platform
      data_source
      uuid
      latest_version
      timestamp
      start_date
      end_date
      versions
      tag
      object_store
    

  
  
    

      0
      medusa
      cgo
      agage
      1200
      ppt
      sio-05
      70m
      cfc11
      surface
      cfc-11
      ...
      not_set
      internal
      48a0e155-3099-479f-9c3b-f4c6078ef479
      v2
      2025-07-03 13:03:31.958090+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
    

  



1 rows × 31 columns






We could also look for details of all the data measured at the Tacolneston
(“TAC”) site using the site keyword:






tac_results = search_surface(site="tac")
tac_results.results













  
    

      
      site
      instrument
      sampling_period
      inlet
      port
      type
      network
      species
      calibration_scale
      long_name
      ...
      platform
      data_source
      uuid
      latest_version
      timestamp
      start_date
      end_date
      versions
      tag
      object_store
    

  
  
    

      0
      tac
      picarro
      3600.0
      54m
      8
      air
      decc
      ch4
      wmo-x2004a
      tacolneston
      ...
      not_set
      internal
      bc0e5994-cd63-42fa-a05b-3f76eff9aa5b
      v2
      2025-07-03 13:03:30.747346+00:00
      2012-07-26 11:23:05+00:00
      2017-12-19 11:25:19+00:00
      {'v1': ['2012-07-26-11:23:05+00:00_2017-12-19-...
      [project1]
      /home/runner/openghg_store/tutorial_store
    

    

      1
      tac
      picarro
      3600.0
      54m
      8
      air
      decc
      co2
      wmo-x2019
      tacolneston
      ...
      not_set
      internal
      da7847fc-bea9-4488-b677-bd2f0bfaac29
      v2
      2025-07-03 13:03:30.772443+00:00
      2012-07-26 11:23:05+00:00
      2017-12-19 11:25:19+00:00
      {'v1': ['2012-07-26-11:23:05+00:00_2017-12-19-...
      [project1]
      /home/runner/openghg_store/tutorial_store
    

    

      2
      tac
      picarro
      3600.0
      100m
      9
      air
      decc
      ch4
      wmo-x2004a
      tacolneston
      ...
      not_set
      internal
      16032361-4c58-4261-93a8-5092f646bb9e
      v2
      2025-07-03 13:03:31.169362+00:00
      2012-07-26 11:04:07+00:00
      2018-01-01 00:23:04+00:00
      {'v1': ['2012-07-26-11:04:07+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
    

    

      3
      tac
      picarro
      3600.0
      100m
      9
      air
      decc
      co2
      wmo-x2019
      tacolneston
      ...
      not_set
      internal
      29424a2c-04eb-46bd-acf3-ea919983f926
      v2
      2025-07-03 13:03:31.193964+00:00
      2012-07-26 11:04:07+00:00
      2018-01-01 00:23:04+00:00
      {'v1': ['2012-07-26-11:04:07+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
    

    

      4
      tac
      picarro
      3600.0
      185m
      10
      air
      decc
      ch4
      wmo-x2004a
      tacolneston
      ...
      not_set
      internal
      d54d3ed2-c3f8-4dc7-a0e0-96ab9c9b348d
      v2
      2025-07-03 13:03:31.550773+00:00
      2013-01-31 00:13:28+00:00
      2018-01-01 00:53:06+00:00
      {'v1': ['2013-01-31-00:13:28+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
    

    

      5
      tac
      picarro
      3600.0
      185m
      10
      air
      decc
      co2
      wmo-x2019
      tacolneston
      ...
      not_set
      internal
      5aeed2b4-995c-4fd6-b45b-87afee41fdf6
      v2
      2025-07-03 13:03:31.573511+00:00
      2013-01-31 00:13:28+00:00
      2018-01-01 00:53:06+00:00
      {'v1': ['2013-01-31-00:13:28+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
    

  



6 rows × 32 columns






For this site you can see this contains details of each of the species
as well as the inlet heights these were measured at.






Searching by tag keyword#


We can also search by the tag keyword when this has been set. Even though the tag
keyword can contain multiple values, this will find all the datasources where the
tag value is included (rather than needing an exact match like the other keywords).


For the “TAC” and “CGO” data we added the “project1” tag and so this data can be found
using this keyword:






results = search_surface(tag="project1")
results.results













  
    

      
      site
      instrument
      sampling_period
      inlet
      port
      type
      network
      species
      calibration_scale
      long_name
      ...
      uuid
      latest_version
      timestamp
      start_date
      end_date
      versions
      tag
      object_store
      units
      species_alt
    

  
  
    

      0
      tac
      picarro
      3600.0
      54m
      8
      air
      decc
      ch4
      wmo-x2004a
      tacolneston
      ...
      bc0e5994-cd63-42fa-a05b-3f76eff9aa5b
      v2
      2025-07-03 13:03:30.747346+00:00
      2012-07-26 11:23:05+00:00
      2017-12-19 11:25:19+00:00
      {'v1': ['2012-07-26-11:23:05+00:00_2017-12-19-...
      [project1]
      /home/runner/openghg_store/tutorial_store
      NaN
      NaN
    

    

      1
      tac
      picarro
      3600.0
      54m
      8
      air
      decc
      co2
      wmo-x2019
      tacolneston
      ...
      da7847fc-bea9-4488-b677-bd2f0bfaac29
      v2
      2025-07-03 13:03:30.772443+00:00
      2012-07-26 11:23:05+00:00
      2017-12-19 11:25:19+00:00
      {'v1': ['2012-07-26-11:23:05+00:00_2017-12-19-...
      [project1]
      /home/runner/openghg_store/tutorial_store
      NaN
      NaN
    

    

      2
      tac
      picarro
      3600.0
      100m
      9
      air
      decc
      ch4
      wmo-x2004a
      tacolneston
      ...
      16032361-4c58-4261-93a8-5092f646bb9e
      v2
      2025-07-03 13:03:31.169362+00:00
      2012-07-26 11:04:07+00:00
      2018-01-01 00:23:04+00:00
      {'v1': ['2012-07-26-11:04:07+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
      NaN
      NaN
    

    

      3
      tac
      picarro
      3600.0
      100m
      9
      air
      decc
      co2
      wmo-x2019
      tacolneston
      ...
      29424a2c-04eb-46bd-acf3-ea919983f926
      v2
      2025-07-03 13:03:31.193964+00:00
      2012-07-26 11:04:07+00:00
      2018-01-01 00:23:04+00:00
      {'v1': ['2012-07-26-11:04:07+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
      NaN
      NaN
    

    

      4
      tac
      picarro
      3600.0
      185m
      10
      air
      decc
      ch4
      wmo-x2004a
      tacolneston
      ...
      d54d3ed2-c3f8-4dc7-a0e0-96ab9c9b348d
      v2
      2025-07-03 13:03:31.550773+00:00
      2013-01-31 00:13:28+00:00
      2018-01-01 00:53:06+00:00
      {'v1': ['2013-01-31-00:13:28+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
      NaN
      NaN
    

    

      5
      tac
      picarro
      3600.0
      185m
      10
      air
      decc
      co2
      wmo-x2019
      tacolneston
      ...
      5aeed2b4-995c-4fd6-b45b-87afee41fdf6
      v2
      2025-07-03 13:03:31.573511+00:00
      2013-01-31 00:13:28+00:00
      2018-01-01 00:53:06+00:00
      {'v1': ['2013-01-31-00:13:28+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
      NaN
      NaN
    

    

      6
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      ch4
      tu1987
      NaN
      ...
      b7a4fd17-f49a-42b8-94f3-224e90beb951
      v2
      2025-07-03 13:03:31.917370+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppb
      NaN
    

    

      7
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      cfc12
      sio-05
      NaN
      ...
      114c1ef9-49af-4290-975d-042ec52db4aa
      v2
      2025-07-03 13:03:31.930679+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppt
      cfc-12
    

    

      8
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      n2o
      sio-16
      NaN
      ...
      34df8481-339c-40d5-a1a9-ede2223c927b
      v2
      2025-07-03 13:03:31.943887+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppb
      NaN
    

    

      9
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      cfc11
      sio-05
      NaN
      ...
      48a0e155-3099-479f-9c3b-f4c6078ef479
      v2
      2025-07-03 13:03:31.958090+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppt
      cfc-11
    

    

      10
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      cfc113
      sio-05
      NaN
      ...
      81174088-e52c-4ff7-84ff-e10818ae9730
      v2
      2025-07-03 13:03:31.971589+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppt
      cfc-113
    

    

      11
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      chcl3
      sio-98
      NaN
      ...
      40a7c65f-d8e0-422b-bd44-8de02336eb1a
      v2
      2025-07-03 13:03:31.985121+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppt
      NaN
    

    

      12
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      ch3ccl3
      sio-05
      NaN
      ...
      362ecbb7-248e-4b2d-a3f1-3d92327c83a9
      v2
      2025-07-03 13:03:31.998745+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppt
      NaN
    

    

      13
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      ccl4
      sio-05
      NaN
      ...
      9d5db302-afee-4ff3-84b5-046205b90303
      v2
      2025-07-03 13:03:32.011938+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppt
      NaN
    

    

      14
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      h2
      mpi-2009
      NaN
      ...
      28749937-a65a-4fa4-b732-a7829a8c43d3
      v2
      2025-07-03 13:03:32.025248+00:00
      2018-01-01 00:30:00+00:00
      2018-12-27 00:02:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-27-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppb
      h2_pdd
    

    

      15
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      co
      csiro-94
      NaN
      ...
      72750e00-7cb7-49b9-a396-b31a6d45504e
      v2
      2025-07-03 13:03:32.038687+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ppb
      NaN
    

    

      16
      cgo
      medusa
      1200
      70m
      NaN
      NaN
      agage
      ne
      na
      NaN
      ...
      bbfd9d33-6474-4144-ac09-28e5c3eb56e1
      v2
      2025-07-03 13:03:32.051621+00:00
      2018-01-01 00:30:00+00:00
      2018-11-19 10:54:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-11-19-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      na
      ne_pdd
    

  



17 rows × 34 columns






For the “CGO” data we also included the “project2” tag and we can find this
data by searching for this:






results = search_surface(tag="project2")
results.results













  
    

      
      instrument
      site
      network
      sampling_period
      units
      calibration_scale
      inlet
      species
      data_type
      inlet_height_magl
      ...
      data_source
      uuid
      latest_version
      timestamp
      start_date
      end_date
      versions
      tag
      object_store
      species_alt
    

  
  
    

      0
      medusa
      cgo
      agage
      1200
      ppb
      tu1987
      70m
      ch4
      surface
      70
      ...
      internal
      b7a4fd17-f49a-42b8-94f3-224e90beb951
      v2
      2025-07-03 13:03:31.917370+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      NaN
    

    

      1
      medusa
      cgo
      agage
      1200
      ppt
      sio-05
      70m
      cfc12
      surface
      70
      ...
      internal
      114c1ef9-49af-4290-975d-042ec52db4aa
      v2
      2025-07-03 13:03:31.930679+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      cfc-12
    

    

      2
      medusa
      cgo
      agage
      1200
      ppb
      sio-16
      70m
      n2o
      surface
      70
      ...
      internal
      34df8481-339c-40d5-a1a9-ede2223c927b
      v2
      2025-07-03 13:03:31.943887+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      NaN
    

    

      3
      medusa
      cgo
      agage
      1200
      ppt
      sio-05
      70m
      cfc11
      surface
      70
      ...
      internal
      48a0e155-3099-479f-9c3b-f4c6078ef479
      v2
      2025-07-03 13:03:31.958090+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      cfc-11
    

    

      4
      medusa
      cgo
      agage
      1200
      ppt
      sio-05
      70m
      cfc113
      surface
      70
      ...
      internal
      81174088-e52c-4ff7-84ff-e10818ae9730
      v2
      2025-07-03 13:03:31.971589+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      cfc-113
    

    

      5
      medusa
      cgo
      agage
      1200
      ppt
      sio-98
      70m
      chcl3
      surface
      70
      ...
      internal
      40a7c65f-d8e0-422b-bd44-8de02336eb1a
      v2
      2025-07-03 13:03:31.985121+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      NaN
    

    

      6
      medusa
      cgo
      agage
      1200
      ppt
      sio-05
      70m
      ch3ccl3
      surface
      70
      ...
      internal
      362ecbb7-248e-4b2d-a3f1-3d92327c83a9
      v2
      2025-07-03 13:03:31.998745+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      NaN
    

    

      7
      medusa
      cgo
      agage
      1200
      ppt
      sio-05
      70m
      ccl4
      surface
      70
      ...
      internal
      9d5db302-afee-4ff3-84b5-046205b90303
      v2
      2025-07-03 13:03:32.011938+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      NaN
    

    

      8
      medusa
      cgo
      agage
      1200
      ppb
      mpi-2009
      70m
      h2
      surface
      70
      ...
      internal
      28749937-a65a-4fa4-b732-a7829a8c43d3
      v2
      2025-07-03 13:03:32.025248+00:00
      2018-01-01 00:30:00+00:00
      2018-12-27 00:02:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-27-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      h2_pdd
    

    

      9
      medusa
      cgo
      agage
      1200
      ppb
      csiro-94
      70m
      co
      surface
      70
      ...
      internal
      72750e00-7cb7-49b9-a396-b31a6d45504e
      v2
      2025-07-03 13:03:32.038687+00:00
      2018-01-01 00:30:00+00:00
      2018-12-31 23:24:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-12-31-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      NaN
    

    

      10
      medusa
      cgo
      agage
      1200
      na
      na
      70m
      ne
      surface
      70
      ...
      internal
      bbfd9d33-6474-4144-ac09-28e5c3eb56e1
      v2
      2025-07-03 13:03:32.051621+00:00
      2018-01-01 00:30:00+00:00
      2018-11-19 10:54:59+00:00
      {'v1': ['2018-01-01-00:30:00+00:00_2018-11-19-...
      [project1, project2]
      /home/runner/openghg_store/tutorial_store
      ne_pdd
    

  



11 rows × 31 columns










Quickly retrieve data#


Say we want to retrieve all the co2 data from Tacolneston, we can
perform perform a search and expect a SearchResults
object to be returned. If no results are found None is returned.






results = search_surface(site="tac", species="co2")
results.results













  
    

      
      site
      instrument
      sampling_period
      inlet
      port
      type
      network
      species
      calibration_scale
      long_name
      ...
      platform
      data_source
      uuid
      latest_version
      timestamp
      start_date
      end_date
      versions
      tag
      object_store
    

  
  
    

      0
      tac
      picarro
      3600.0
      54m
      8
      air
      decc
      co2
      wmo-x2019
      tacolneston
      ...
      not_set
      internal
      da7847fc-bea9-4488-b677-bd2f0bfaac29
      v2
      2025-07-03 13:03:30.772443+00:00
      2012-07-26 11:23:05+00:00
      2017-12-19 11:25:19+00:00
      {'v1': ['2012-07-26-11:23:05+00:00_2017-12-19-...
      [project1]
      /home/runner/openghg_store/tutorial_store
    

    

      1
      tac
      picarro
      3600.0
      100m
      9
      air
      decc
      co2
      wmo-x2019
      tacolneston
      ...
      not_set
      internal
      29424a2c-04eb-46bd-acf3-ea919983f926
      v2
      2025-07-03 13:03:31.193964+00:00
      2012-07-26 11:04:07+00:00
      2018-01-01 00:23:04+00:00
      {'v1': ['2012-07-26-11:04:07+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
    

    

      2
      tac
      picarro
      3600.0
      185m
      10
      air
      decc
      co2
      wmo-x2019
      tacolneston
      ...
      not_set
      internal
      5aeed2b4-995c-4fd6-b45b-87afee41fdf6
      v2
      2025-07-03 13:03:31.573511+00:00
      2013-01-31 00:13:28+00:00
      2018-01-01 00:53:06+00:00
      {'v1': ['2013-01-31-00:13:28+00:00_2018-01-01-...
      [project1]
      /home/runner/openghg_store/tutorial_store
    

  



3 rows × 32 columns






We can retrieve either some or all of the data easily using the
retrieve function.






inlet_54m_data = results.retrieve(inlet="54m")
inlet_54m_data









<openghg.dataobjects._obsdata.ObsData at 0x7f2980377440>









Or we can retrieve all of the data and get a list of ObsData
objects.






all_co2_data = results.retrieve_all()
all_co2_data









[<openghg.dataobjects._obsdata.ObsData at 0x7f29702f0080>,
 <openghg.dataobjects._obsdata.ObsData at 0x7f297087ef00>,
 <openghg.dataobjects._obsdata.ObsData at 0x7f2970002ab0>]















3. Retrieving data#


To retrieve the standardised data from the object store there are
several functions we can use which depend on the type of data we want to
access.


To access the surface data we have added so far we can use the
get_obs_surface function and pass keywords for the site code,
species and inlet height to retrieve our data. Using get_* functions will only allow one set of data to be returned and will give details if this is not the case.


In this case we want to extract the carbon dioxide (“co2”) data from the
Tacolneston data (“TAC”) site measured at the “185m” inlet:






from openghg.retrieve import get_obs_surface

co2_data = get_obs_surface(site="tac", species="co2", inlet="185m")













If we view our returned obs_data variable this will contain:


    

  • data - The standardised data (accessed using
e.g. obs_data.data). This is returned as an xarray
Dataset.
    • 

  • metadata - The associated metadata (accessed using
e.g. obs_data.metadata).
    • 







co2_data









<openghg.dataobjects._obsdata.ObsData at 0x7f2980b51250>













co2_data.data

























<xarray.Dataset> Size: 1MB
Dimensions:                    (time: 39114)
Coordinates:
  * time                       (time) datetime64[ns] 313kB 2013-01-31T00:13:2...
Data variables:
    mf                         (time) float64 313kB dask.array<chunksize=(19557,), meta=np.ndarray>
    mf_number_of_observations  (time) float64 313kB dask.array<chunksize=(19557,), meta=np.ndarray>
    mf_variability             (time) float64 313kB dask.array<chunksize=(19557,), meta=np.ndarray>
Attributes: (12/28)
    Conventions:           CF-1.8
    comment:               Cavity ring-down measurements. Output from GCWerks
    conditions_of_use:     Ensure that you contact the data owner at the outs...
    data_owner:            Simon O'Doherty
    data_owner_email:      s.odoherty@bristol.ac.uk
    data_source:           internal
    ...                    ...
    station_latitude:      52.51882
    station_long_name:     Tacolneston Tower, UK
    station_longitude:     1.1387
    tag:                   project1
    type:                  air
    scale:                 WMO-X2019








co2_data.metadata









{'data_type': 'surface',
 'site': 'tac',
 'instrument': 'picarro',
 'sampling_period': '3600.0',
 'inlet': '185m',
 'port': '10',
 'type': 'air',
 'network': 'decc',
 'species': 'co2',
 'calibration_scale': 'WMO-X2019',
 'long_name': 'tacolneston',
 'inlet_height_magl': '185',
 'data_owner': "Simon O'Doherty",
 'data_owner_email': 's.odoherty@bristol.ac.uk',
 'station_longitude': '1.1387',
 'station_latitude': '52.51882',
 'station_long_name': 'Tacolneston Tower, UK',
 'station_height_masl': 64,
 'source_format': 'CRDS',
 'data_level': 'not_set',
 'data_sublevel': 'not_set',
 'dataset_source': 'not_set',
 'platform': 'not_set',
 'data_source': 'internal',
 'uuid': '5aeed2b4-995c-4fd6-b45b-87afee41fdf6',
 'latest_version': 'v2',
 'timestamp': '2025-07-03 13:03:31.573511+00:00',
 'start_date': '2013-01-31 00:13:28+00:00',
 'end_date': '2018-01-01 00:53:06+00:00',
 'versions': {'v1': ['2013-01-31-00:13:28+00:00_2018-01-01-00:53:06+00:00'],
  'v2': ['2013-01-31-00:13:28+00:00_2018-01-01-00:53:06+00:00']},
 'tag': 'project1',
 'object_store': '/home/runner/openghg_store/tutorial_store',
 'Conventions': 'CF-1.8',
 'comment': 'Cavity ring-down measurements. Output from GCWerks',
 'conditions_of_use': 'Ensure that you contact the data owner at the outset of your project.',
 'file_created': '2025-07-03 13:03:31.524781+00:00',
 'processed_by': 'OpenGHG_Cloud',
 'sampling_period_unit': 's',
 'source': 'In situ measurements of air',
 'scale': 'WMO-X2019'}









We can now make a simple plot using the plot_timeseries method of
the ObsData object.




NOTE: the plot created below may not show up on the online
documentation version of this notebook.








co2_data.plot_timeseries()

















You can also pass any of title, xlabel, ylabel and units
to the plot_timeseries function to modify the labels.






4. Cleanup#


If you’re finished with the data in this tutorial you can cleanup the
tutorial object store using the clear_tutorial_store function.






from openghg.tutorial import clear_tutorial_store

















clear_tutorial_store()