The functions and methods documented in this section are the internal workings of the OpenGHG library. They are subject to change without warning due to the early stages of development of the project.
Normal users should not use any of the functions shown here directly as they may be removed or their functionality may change.
This provides the functionality required by all data storage and processing classes, namely the saving, retrieval and loading of data from the object store.
Base class which the other core processing modules inherit
These classes are used for the processing of data by the
ObsSurface processing class.
For processing data from Cranfield
For processing data from CRDS (cavity ring-down spectroscopy) data from the DECC network.
For processing data from the EUROCOM network
For processing data in the form expected by the GCWERKS package
For processing data from the ICOS network
For processing data from the NOAA network
For processing data from the Thames Barrier measurement sites
The Datasource is the smallest data provider within the OpenGHG topology. A Datasource represents a data provider such as an instrument measuring a specific gas at a specific height at a specific site. For an instrument measuring three gas species at an inlet height of 100m at a site we would have three Datasources.
Handles the storage of data, metadata and version information for measurements
These functions handle the storage of data in the object store, in JSON or binary format. Each object and piece of data in the
object store is stored at a specific
key, which can be thought of as the address of the data. The data is stored
bucket which in the cloud is a section of the OpenGHG object store. Locally a
bucket is just a normal
directory in the user’s filesystem specific by the
OPENGHG_PATH environment variable.
Delete an object in the store
Check if an object exists at that key
Get path to bucket
Get path to local bucket
Get object at given key
Get object from JSON
Set data at a key from a given filepath
Set data at a key from JSON
This module contains all the helper functions used throughout OpenGHG.
These are used to export data to a format readable by the OpenGHG data dashboard.
Export timeseries data to JSON
Export mobile observations data to JSON
These handle hashing of data (usually with SHA1)
Calculate the SHA1 hash of a file
Calculate the SHA1 hash of a UTF-8 encoded string
String cleaning and formatting functions
Return a lowercase cleaned string
Converts a string to lowercase
Helpers to deal with all things datetime.
Create a Timestamp with a UTC timezone
Create a timezone aware timestamp for now
Create a timezone aware timestamp for the UNIX epoch (1970-01-01)
Create a daterange from two timestamp strings
Convert a daterange to string
Create a daterange string from two timestamps or strings
Create a pandas DatetimeIndex from two timestamps
Check if two dateranges overlap
Combine a list of dateranges
Split a daterange string to the component start and end Timestamps
Finds the closest daterange in a list of dateranges
Check if the passed daterange is valid
Find the gaps in a list of dateranges
Removes overlapping dates from to_trim
Checks if one of the passed dateranges contains the other, if so, then split the larger daterange into three sections.
Checks if one daterange contains another
Make sure the daterange is correct and return tzaware daterange.
Check if the given value is NaN, is so return an NA string
Check if the passed string is a valid date or not, if not returns NA
Our own personal
Return a zip of an iterable where a is the iterable and b is the iterable advanced one step.
Checks that all values in an iterable object are the same
These perform checks to ensure data processed for each site is correct
Check if the passed site is a valid one
Check if the passed site has more than one inlet