Data access & code

Model simulations

The main model simulations conducted by PRIMAVERA groups follow the CMIP6 HighResMIP protocol. The model outputs are currently being made available on the CMIP ESGF (Earth System Grid Federation).

The project itself has all these simulations available on the CEDA-JASMIN platform, mainly on tape. We use the PRIMAVERA DMT (Data Management Tool) to select and extract the data from tape to disk. This tool can also be used to query which CMOR-formatted variables are available from our simulations. The DMT is available for users to query the available data. ​View the DMT user guide

To avoid having to read all of the user guide, please go to https://prima-dm1.jasmin.ac.uk/received_data_quick_query/ to see what data has been received. Hopefully the search is fairly intuitive. If not, then please let us know and we will work to improve it, or to improve its documentation.

If you would like to collaborate with PRIMAVERA and use CEDA-JASMIN with us to conduct complementary analysis, then please contact us for details. A summary of the required technical steps is available below.

Further information on coordinated analysis of the HighResMIP simulations is available.

Data access for CLIVAR and other groups

Users of PRIMAVERA data (e.g. CLIVAR collaborators and others) should follow the instructions at http://help.jasmin.ac.uk/article/189-get-started-with-jasmin to get access to the JASMIN facilities:

  • At step 4, applying for the jasmin-login role, you should include the email addresses of two UK-based PRIMAVERA project member who can provide you with a reference to support your application. Hence you will need to speak with us before you apply.
  • To access the data you will need to register for a CEDA account at ​http://www.ceda.ac.uk/ and link this to your JASMIN account at ​https://accounts.jasmin.ac.uk/account/profile/ to get access to the CEDA archives.

The data that is available can be queried using the DMT (Data Management Tool) available at https://prima-dm1.jasmin.ac.uk/. Most people will find the Variables Received Query (https://prima-dm1.jasmin.ac.uk/received_data_quick_query/) is the best way to search through the available data. Data will have an “Online Status” of online if all of that data is currently available on disk, and offline or partial if some data is only available on tape. If data is only available on tape then users can use the request retrieval buttons to the right of the table and the “Create Retrieval Request” button at the bottom of the page to request that data is restored from tape to disk. Restoring data from tape to disk may take several days depending upon how busy the tape system is.  

Anyone can use the DMT’s queries to search for available data. The complete user guide for the DMT is available from https://docs.google.com/document/d/1rz4KB3xFUiqsWsCd85Kv4HIBOPM5e7n8QHa6urxWdXY/

The PRIMAVERA group workspaces at JASMIN are no longer available and their storage has been handed back for use by other projects. All of the data produced is available from the CEDA archives instead and can be accessed by following the instructions above for CEDA archive access. The paths in the DMT show the path to the data in the CEDA archives. Users wishing to store intermediate files should use JASMIN temporary storage.

Code resources

We are building up a library of useful utilities within the project which may have a wider application. Please see the github site: https://github.com/PRIMAVERA-H2020

We have a variety of packages of code, including:

  • HighResMIP-storm_reader: Python code to read the CMOR-like storm files that will become available on the CEDA archives
  • HighResMIP-extreme_indices: Python code using the ICCLIM package to calculate climate extreme indices
  • HighResMIP-futureSSTSeaice: code base used to generate the future SST and sea-ice forcing for the HighResMIP experiment highresSST-future out to 2050

Coming soon: standard code used to calculate other model indices (such as AMO, AMM, El Nino), as well as code to calculate power spectra and the like, to be used in forthcoming manuscripts from the project.

Data resources

Full data outputs from the models (as per the CMIP6 HighResMIP data request specification) are now being published on the ESGF.

Tropical and extra-tropical cyclones

Tracking the model simulations (using TRACK and TempestExtremes algorithms to far) for both tropical and extra-tropical storms in ongoing. This data is available at CEDA-JASMIN and from the CEDA archives at https://catalogue.ceda.ac.uk/uuid/e82a62d926d7448696a2b60c1925f811.

The list of models that have been tracked can be seen at this Google doc.

Climate extremes indices

These are being calculated using the ICCLIM software (see above). Michael Wehner has made these available via a server at Berkeley: http://portal.nersc.gov/archive/home/projects/cascade/www/ETCCDI/

Tutorial - Accessing the data from the ESGF

The Earh System Grid Federation (ESGF) was developed to disseminate data from all CMIP models and experiments and so can be confusing for first time users. This is brief tutorial to help you get started.

The ESGF is a distributed grid of data nodes. There are many nodes around the world. You can use any node to browse through the metadata published to any of the other nodes. When you download a file your download will automatically be redirected to the node that holds the file.

All PRIMAVERA data was published to the HighResMIP activity and is available from the PRIMAVERA data search on CEDA's ESGF node. Many other institutes have also contributed data to HighResMIP and so you will also see non-PRIMAVERA data in the ESGF search. You can see the names of the PRIMAVERA models on our Modelling page.

Once you have chosen the HighResMIP activity then setting the "Experiment" is a useful way to further subset the data. You can find details of the PRIMAVERA experiments on Our Simulations page. To chose individual variables then you can use the "Variable" or "CF Standard Name" searches on the left-hand side of the ESGF search page. The "Variable" names may not be intuitive at first. The "CF Standard Name" is probably more intuitive. The CF Standard names are defined on CF Conventions website. An example variable is "tas" which has a CF Standard Name of "air_temperature".

A typical dataset on the ESGF search looks like:

ESGF entry for a typical dataset

The "THREDDS Catalog" link will allow you to download individual files and requires no registration or account. However, some datasets can be split over many files and so this could be time consuming.

The "WGET Script" link will download a BASH script that when run with the -H option will download the chosen dataset. BASH scripts can be run natively on Mac, Linux and Unix operating systems. Various BASH shells can be installed for Windows. You will also need the wget software installed,  which comes with many operating systems but is available for all. To use the WGET Script you will need to register for a free CEDA account at http://archive.ceda.ac.uk/. When running the WGET script you will be prompted to enter your CEDA OpenID and CEDA password.