DreqPy (Data Request Python API)

CMIP Phase 6 Data Request Python Interface created by Martin Juckes

IPython NoteBook created by Martin Schupfner, DKRZ

Reference: Martin Juckes, NCAS BADC (2016) -

dreqPy (Data Request Python API) & dreqPy User's Guide



Download this website.

Synopsis

0 - Basic Information

1 - Basic Imports

2 - dq.coll Examples

3 - dq.inx Examples

4 - Presentation of dq.coll sections

5 - Advanced Example

6 - Data Request Volume Estimate

7 - Additional Functions

8 - Official Web Tools

0 - Basic Information

In [3]:
display(Image(filename='files/Folie2.PNG'))
In [4]:
display(Image(filename='files/Folie3.PNG'))
In [5]:
display(Image(filename='files/Folie4.PNG'))

1 - Basic Imports

In [6]:
#!/usr/bin/python
# -*- coding: utf-8 -*-

# IPython NoteBook created by Martin Schupfner, DKRZ
# Reference: Martin Juckes 2016 - 
#            dreqPy (Data Request Python API) & dreqPy User's Guide

from dreqPy import dreq

print "Using DreqPy (Data Request Python API) in version %s"\
    % str(dreq.version)

# Initialisation
dq = dreq.loadDreq()
Using DreqPy (Data Request Python API) in version 01.beta.42

2 - dq.coll Examples

In [7]:
# dq.coll
# Content Object dq.coll is a dictionary containing the data request sections,
#   with 3 elements (named tuples) for each section:
# - header :  named tuple with info such as title, lable, etc.
# - attDefn:  dictionary containing record attribute definitions
# - items  :  list of records

# Print all entries of dq.coll
print "dq.coll Entries:\n", ", ".join(dq.coll.keys())
dq.coll Entries:
requestItem, grids, spatialShape, miptable, requestVar, __main__, temporalShape, __sect__, modelConfig, experiment, var, mip, __core__, tags, CMORvar, tableSection, varChoice, exptgroup, varRelLnk, remarks, varRelations, cellMethods, structure, requestVarGroup, varChoiceLinkC, requestLink, varChoiceLinkR, objectiveLink, objective, timeSlice, standardname
In [8]:
# header content (Example MIP Variable):
print ".header content Example 'var':\n------------------------------"

for name in dq.coll['var'].header._fields: print "%-15s : %s" \
    %(name, getattr(dq.coll['var'].header, name))
.header content Example 'var':
------------------------------
tag             : table
label           : var
title           : 1.2 MIP Variable
id              : var
itemLabelMode   : def
level           : 0
maxOccurs       : 1
labUnique       : No
uid             : SECTION:var
In [9]:
# attDefn content (Example MIP Variable):
print ".attDefn content Example 'var':\n-------------------------------"

for key in dq.coll['var'].attDefn.keys(): 
    print "%-15s : %s" %(key, dq.coll['var'].attDefn[key])
.attDefn content Example 'var':
-------------------------------
procnote        : Item <X.1 Core Attributes>: [procnote] Processing notes
description     : Item <X.1 Core Attributes>: [description] Record Description
title           : Item <X.1 Core Attributes>: [title] Long name
prov            : Item <X.1 Core Attributes>: [prov] Provenance
unid            : Item <X.1 Core Attributes>: [unid] Link to Units section
label           : Item <X.1 Core Attributes>: [label] Variable Name
provmip         : Item <X.1 Core Attributes>: [provmip] MIP defining this variables
sn              : Item <X.1 Core Attributes>: [sn] CF Standard Name
units           : Item <X.1 Core Attributes>: [units] Units
procComment     : Item <X.1 Core Attributes>: [procComment] Processing comments
uid             : Item <X.1 Core Attributes>: [uid] Record identifier
In [10]:
# items content (Example MIP Variable):
print ".items content Example 'var':\n-----------------------------"

for key in dq.coll['var'].attDefn.keys(): 
    print "%-15s : %s" %(key, getattr(dq.coll['var'].items[0], key))
.items content Example 'var':
-----------------------------
procnote        : ('',)
description     : 
title           : Ambient Aerosol Absorption Optical Thickness at 550 nm
prov            : CMIP5_aero
unid            : fd6fef50-3468-11e6-ba71-5404a60d96b5
label           : abs550aer
provmip         : CMIP5
sn              : atmosphere_absorption_optical_thickness_due_to_ambient_aerosol
units           : 1.0
procComment     : 
uid             : 0baf6a333b91c4db341b515c28cd2c05

3 - dq.inx Examples

In [11]:
# Index dq.inx is a simple lookup table
#   dq.inx.uid[UID] returns the record corresponding to 'UID'

#   Example from above:
item = dq.inx.uid['0baf6a333b91c4db341b515c28cd2c05']

print item
print "Label:", item.label
print "Title:", item.title
print "Units:", item.units
Item <1.2 MIP Variable>: [abs550aer] Ambient Aerosol Absorption Optical Thickness at 550 nm
Label: abs550aer
Title: Ambient Aerosol Absorption Optical Thickness at 550 nm
Units: 1.0
In [12]:
#   dq.inx.iref_by_uid[UID]
#    gives a list of IDs of objects linked to 'UID',
#    as a tuple (SECT, ID)

#   Example from above:
id_list = dq.inx.iref_by_uid['0baf6a333b91c4db341b515c28cd2c05']

for ID in id_list:     
    print "#  Section: %-10s\n   UID: %15s\n   %s\n"\
    %(ID[0], ID[1], dq.inx.uid[ID[1]])
#  Section: vid       
   UID: 19bebf2a-81b1-11e6-92de-ac72891c3257
   Item <1.3 CMOR Variable>: [abs550aer] ambient aerosol absorption optical thickness at 550 nm

#  Section: tid       
   UID: atmosphere_absorption_optical_thickness_due_to_ambient_aerosol
   Item <3.8 Remarks about other items>: [__unset__] __unset__

In [13]:
#   dq.inx.iref_by_sect[UID].a[SECT] 
#    gives a list of IDs of 'SECT' linked to 'UID'

#   Example from above:
id_list = dq.inx.iref_by_sect['0baf6a333b91c4db341b515c28cd2c05'].a['CMORvar']

for ID in id_list: 
    item = dq.inx.uid[ID]
    print "%15s | %10s | %s" \
    %(item.label, item.mipTable, item.vid)
      abs550aer | aermonthly | 0baf6a333b91c4db341b515c28cd2c05
In [14]:
# The same can be achieved using dq.coll, though:

#   Example from above:
CMORvar_list = [ item for item in dq.coll['CMORvar'].items \
           if item.vid=='0baf6a333b91c4db341b515c28cd2c05']

for item in CMORvar_list:     
    print "%15s | %10s | %s" \
    %(item.label, item.mipTable, item.vid)
      abs550aer | aermonthly | 0baf6a333b91c4db341b515c28cd2c05

4 - Presentation of dq.coll Sections

Overview over all dq.coll sections and their items' attributes, as well as examples.

In [15]:
from IPython.display import Image, display

# Display Structure of DreqPy
display(Image(filename='files/DreqPyStructure.png'))
In [16]:
def showSect(sect, desc=False):
    """
    Print the Content of dq.coll's sections
    Arg: sect - str     - section key
         desc - boolean - also print description 
                          (Default: False)
    """
    
    # print header title
    print dq.coll[sect].header.title
    print "-"*len(dq.coll[sect].header.title)
    
    # print Section name and description
    for subsect in dq.coll[sect].attDefn.keys():
        if desc==True:
            print "# %15s  |  %s%s" \
            %(subsect, dq.coll[sect].attDefn[subsect].title, \
            "\n"+dq.coll[sect].attDefn[subsect].description if \
            (str(dq.coll[sect].attDefn[subsect].description)!="" or \
            subsect in str(dq.coll[sect].attDefn[subsect].description)) \
            else "")
        else:
            print "# %15s  |  %s" \
            %(subsect, dq.coll[sect].attDefn[subsect].title)            
            
    # print Example of first item in list
    print "-------\nExample\n-------"
    for subsect in dq.coll[sect].attDefn.keys():
            print "# %15s  |  %s"\
            %(subsect, getattr(dq.coll[sect].items[0], subsect) if \
             subsect not in str(getattr(dq.coll[sect].items[0], subsect)) \
             else "")
In [17]:
# MIPs
print "Chicken or egg? MIP or Objective?\n"\
      "The CMIP endorsed MIP (Model Intercomparison Project) is linked to scientific objectives."
display(Image(filename='files/DreqPyStructure_1.png'))
showSect('mip')
#showSect('mip', desc=True) #Activate for additional info if there is any
Chicken or egg? MIP or Objective?
The CMIP endorsed MIP (Model Intercomparison Project) is linked to scientific objectives.
1.1 Model Intercomparison Project
---------------------------------
#             url  |  Project Home Page
#     description  |  Description of the Model Intercomparison Project
#           title  |  MIP title
#             uid  |  Record identifier
#           label  |  MIP short name
-------
Example
-------
#             url  |  http://www.wcrp-climate.org/wgcm-cmip/wgcm-cmip6
#     description  |  __unset__
#           title  |  Coupled Model Intercomparison Project, Phase 6
#             uid  |  CMIP6
#           label  |  CMIP6
In [18]:
# Objective
showSect('objective')
#showSect('objective', desc=True) #Activate for additional info if there is any
1.6 Scientific objectives
-------------------------
#             mip  |  Endorsed MIP
#     description  |  Description
#           title  |  Record Title
#             uid  |  Record identifier
#           label  |  Record Label
-------
Example
-------
#             mip  |  C4MIP
#     description  |  quantify and understand land and ocean carbon cycle feedbacks with climate
#           title  |  climate-carbon cycle feedbacks
#             uid  |  dc90cf0a-8308-11e5-b787-ac72891c3257
#           label  |  CarbonFeedbacks
In [19]:
# Experiments
print "MIPs may define experiments or solely request data from experiments."
print "Experiments are organised in experiment groups."
display(Image(filename='files/DreqPyStructure_2.png'))
showSect('experiment')
#showSect('experiment', desc=True) #Activate for additional info if there is any
MIPs may define experiments or solely request data from experiments.
Experiments are organised in experiment groups.
1.5 Experiments
---------------
#          nstart  |  Number of start dates
#             yps  |  Years per simulation
#          starty  |  Start year
#     description  |  Description
#           title  |  Record Title
#            endy  |  End year
#            ensz  |  Ensemble size
#           label  |  Record Label
#            egid  |  Identifier for experiment group
#            tier  |  Tier of experiment
#             mip  |  MIP defining experiment
#            ntot  |  Total number of years
#            mcfg  |  Model category
#         comment  |  Comment
#             uid  |  Record identifier
-------
Example
-------
#          nstart  |  1
#             yps  |  85
#          starty  |  2015.0
#     description  |  Emission-driven future scenario simulation,  biogeochemically-coupled
#           title  |  __unset__
#            endy  |  2100.0
#            ensz  |  [1]
#           label  |  esmssp5-85bgc
#            egid  |  b111fe1c-aab7-11e6-9fd2-ac72891c3257
#            tier  |  [2]
#             mip  |  C4MIP
#            ntot  |  85
#            mcfg  |  
#         comment  |  
#             uid  |  b112010a-aab7-11e6-9fd2-ac72891c3257
In [20]:
# Experiment Groups
showSect('exptgroup')
#showSect('exptgroup', desc=True) #Activate for additional info if there is any
1.9 Experiment Group
--------------------
#         tierMin  |  Minimum tier of experiments in group
#           title  |  Record Title
#             uid  |  Record identifier
#            ntot  |  Total number of years
#           label  |  Record Label
-------
Example
-------
#         tierMin  |  1
#           title  |  __unset__
#             uid  |  b111e90e-aab7-11e6-9fd2-ac72891c3257
#            ntot  |  140
#           label  |  C4mip1
In [21]:
# Request Item
print "The request items build up the data request."
print "They are linked to (or: requested by) either MIPs, experiments or experiment groups."
print "Request items hold information about the time slice of the requested variables they include\n"\
      "as well as the possibility to set their priority and tier."
display(Image(filename='files/DreqPyStructure_3.png'))
showSect('requestItem')
#showSect('requestItem', desc=True) #Activate for additional info if there is any
The request items build up the data request.
They are linked to (or: requested by) either MIPs, experiments or experiment groups.
Request items hold information about the time slice of the requested variables they include
as well as the possibility to set their priority and tier.
3.2 Request Item: specifying the number of years for an experiment
------------------------------------------------------------------
#          treset  |  Option to override tier set for experiment(s)
#          nexmax  |  Maximum number of experiments requested.
#             uid  |  Record Identifier
#           title  |  Record Title
#          nenmax  |  Number of ensemble members requested.
#           nymax  |  Number of years requested.
#            expt  |  Name of experiment or group of experiments
#              ny  |  Default number of years.
#          preset  |  Option to override priority set in each variable group
#          tslice  |  Selection of years from experiment
#            rlid  |  Identifier of corresponding requestLink
#             tab  |  Redundant?
#            esid  |  Identifier experiment(s): a link to an experiment, an experiment group or a MIP
#             mip  |  The MIP making the request. 
#           label  |  Record Label
#     esidComment  |  Comment on experiment(s) linked to.
-------
Example
-------
#          treset  |  
#          nexmax  |  -999
#             uid  |  b7095ea2-aaa6-11e6-860b-ac72891c3257
#           title  |  C4MIP, C4MIP-Basic, historical-ext
#          nenmax  |  -1
#           nymax  |  -1.0
#            expt  |  historical-ext
#              ny  |  160
#          preset  |  -1
#          tslice  |  
#            rlid  |  efc0d8f0-5629-11e6-9079-ac72891c3257__00
#             tab  |  C4MIP-Basic
#            esid  |  b1148498-aab7-11e6-9fd2-ac72891c3257
#             mip  |  C4MIP
#           label  |  C4mipC4mipBasic
#     esidComment  |  Experiment historical-ext
In [22]:
# Time Slice
showSect('timeSlice')
#showSect('timeSlice', desc=True) #Activate for additional info if there is any
3.11 Time Slices for Output Requests
------------------------------------
#             end  |  End year
#             uid  |  Unique identifier
#           title  |  Record Title
#          nyears  |  Total number of years
#           label  |  Record Label
#           start  |  Start year
#            step  |  Step (years)
#       startList  |  Optional list of start times.
#           child  |  Child experiment
#        sliceLen  |  Length of slice
#            type  |  Type of time slice
#    sliceLenUnit  |  Units of slice length
#     description  |  Description
-------
Example
-------
#             end  |  2020
#             uid  |  _slice_DAMIP42
#           title  |  DAMIP 42 year
#          nyears  |  42.0
#           label  |  DAMIP42
#           start  |  1979
#            step  |  
#       startList  |  
#           child  |  __unset__
#        sliceLen  |  
#            type  |  simpleRange
#    sliceLenUnit  |  
#     description  |  
In [23]:
# Request Variable Group
print "The request variable groups are defined by MIPs."
display(Image(filename='files/DreqPyStructure_4.png'))
showSect('requestVarGroup')
#showSect('requestVarGroup', desc=True) #Activate for additional info if there is any
The request variable groups are defined by MIPs.
3.1 Request variable group: a collection of request variables
-------------------------------------------------------------
#         refNote  |  Reference Note
#             uid  |  Record Identifier
#           title  |  Record Title
#           label  |  Record Label
#             mip  |  Endorsed MIP defining the variable group
#             ref  |  Reference
-------
Example
-------
#         refNote  |  __unset__
#             uid  |  6d99677c-5979-11e6-8fd9-ac72891c3257
#           title  |  VIACSAB: CSP.4
#           label  |  CSP-4
#             mip  |  VIACSAB
#             ref  |  __unset__
In [24]:
# Request Link
print "Each request item is linked (via request link) to a request variable group."
print "Several request items can link to the same request variable group\n"\
      "for a different set of experiments, time slices, priorities, etc."
print "Of course a request variable group can be requested by MIPs that did not define it."
display(Image(filename='files/DreqPyStructure_5.png'))
showSect('requestLink')
#showSect('requestLink', desc=True) #Activate for additional info if there is any
Each request item is linked (via request link) to a request variable group.
Several request items can link to the same request variable group
for a different set of experiments, time slices, priorities, etc.
Of course a request variable group can be requested by MIPs that did not define it.
3.3 Request link: linking a set of variables and a set of experiments
---------------------------------------------------------------------
#         comment  |  Comment
#         refNote  |  Note on reference
#             uid  |  Record Identifier
#            opar  |  parameter associated with *opt*
#           title  |  Record Title
#             opt  |  option for selecting a subset of variables
#           label  |  Record Label
#            grid  |  Grid options
#             tab  |  Redundant
#       objective  |  Science objectives associated with this request
#             mip  |  Endorsed MIP requesting the data
#             ref  |  Reference
#           refid  |  reference to a request Variable Group
#         gridreq  |  Grid option constraints
-------
Example
-------
#         comment  |  
#         refNote  |  
#             uid  |  26d3f07c-7c16-11e6-8a5b-ac72891c3257__02
#            opar  |  
#           title  |  day-oth
#             opt  |  list
#           label  |  __unset__
#            grid  |  
#             tab  |  day-oth
#       objective  |  
#             mip  |  CORDEX
#             ref  |  
#           refid  |  26d3f07c-7c16-11e6-8a5b-ac72891c3257
#         gridreq  |  
In [25]:
# Objective Link
print "Request items are linked to scientific objectives (via objective link and request link)."
display(Image(filename='files/DreqPyStructure_6.png'))
showSect('objectiveLink')
#showSect('objectiveLink', desc=True) #Activate for additional info if there is any
Request items are linked to scientific objectives (via objective link and request link).
3.7 Link between scientific objectives and requests
---------------------------------------------------
#             rid  |  Identifier for a request link
#           title  |  Record Title
#             uid  |  Record identifier
#             oid  |  Identifier for a scientific objective
#           label  |  Record Label
-------
Example
-------
#             rid  |  efc0e43a-5629-11e6-9079-ac72891c3257__00
#           title  |  __unset__
#             uid  |  8817b79e-aab8-11e6-b8d4-ac72891c3257
#             oid  |  dc919dcc-8308-11e5-b787-ac72891c3257
#           label  |  C4MIP
In [26]:
# Request Variable
print "A request variable group holds a list of request variables."
print "Request variables are basically links to CMOR variables with an\n"\
      "additional information about the CMOR variables' priority."
display(Image(filename='files/DreqPyStructure_7.png'))
showSect('requestVar')
#showSect('requestVar', desc=True) #Activate for additional info if there is any
A request variable group holds a list of request variables.
Request variables are basically links to CMOR variables with an
additional information about the CMOR variables' priority.
1.4 Request variable (carrying priority and link to group)
----------------------------------------------------------
#             uid  |  Record identifier
#             vid  |  Identifier for MIP Output Variable
#           title  |  Record Title
#           label  |  Record Label
#        priority  |  Variable priority
#            vgid  |  Identifier for Variable Group
#             mip  |  Endorsed MIP
-------
Example
-------
#             uid  |  8709b6f4-aab8-11e6-b8d4-ac72891c3257
#             vid  |  19bee41e-81b1-11e6-92de-ac72891c3257
#           title  |  emioa ((isd.005))
#           label  |  emioa
#        priority  |  1
#            vgid  |  ef12fa96-5629-11e6-9079-ac72891c3257
#             mip  |  AerChemMIP
In [27]:
# CMOR Variable
showSect('CMORvar')
#showSect('CMORvar', desc=True) #Activate for additional info if there is any
1.3 CMOR Variable
-----------------
#         shuffle  |  Shuffle: NetCDF compression parameter
#             uid  |  Record Identifier
#             vid  |  MIP Variable
#       valid_min  |  Minimum expected value for this variable.
#       frequency  |  Frequency of time steps to be archived.
# ok_max_mean_abs  |  Maximum expected value of the mean absolute value at each point in time
#           title  |  Long name
#        rowIndex  |  Row index of entry in source sheet
#            prov  |  Provenance
#            stid  |  Link to a record specifying the structure of the variable (dimensions and associated variable attributes).
#        mipTable  |  The MIP table: each table identifies a collection of variables
#           label  |  CMOR Variable Name
#            mtid  |  Link to MIP table record
#        subGroup  |  Sub-group of variables in a table
#            type  |  Data value type, e.g. float or double
#   deflate_level  |  Deflate Level: NetCDF compression parameter
#     description  |  Description
#      processing  |  Processing notes
#         deflate  |  Deflate: NetCDF compression parameter
# mipTableSection  |  Section of a table
#        provNote  |  Provenance Note
# ok_min_mean_abs  |  Minimum expected value of the mean absolute value at each point in time
# defaultPriority  |  Indicative priority for this parameter, which is over-ruled by the requestVar priority setting, but provides a reference for organisation of the CMORvariables
#  modeling_realm  |  Modeling Realm
#        positive  |  CMOR Directive Positive
#       valid_max  |  Maximum expected value for this variable.
-------
Example
-------
#         shuffle  |  
#             uid  |  baa123da-e5dd-11e5-8482-ac72891c3257
#             vid  |  1333394a296e7f8af6c9bad15cb9778d
#       valid_min  |  
#       frequency  |  mon
# ok_max_mean_abs  |  
#           title  |  Rate of Change of Net Dissolved Inorganic Carbon
#        rowIndex  |  125
#            prov  |  Omon ((isd.003))
#            stid  |  f9475c26-80ba-11e6-ab6e-5404a60d96b5
#        mipTable  |  Omon
#           label  |  fddtdic
#            mtid  |  MIPtable::Omon
#        subGroup  |  Omon_oth
#            type  |  real
#   deflate_level  |  
#     description  |  
#      processing  |  integral over upper 100 m only.
#         deflate  |  
# mipTableSection  |  Omon_oth
#        provNote  |  Omon_oth
# ok_min_mean_abs  |  
# defaultPriority  |  3
#  modeling_realm  |  ocnBgchem
#        positive  |  
#       valid_max  |  
In [28]:
# MIP Variable
print "MIP variables are defined by a MIP and linked to CMOR variables."
print "They hold information like the variables' unit, etc."
display(Image(filename='files/DreqPyStructure_8.png'))
showSect('var')
#showSect('var', desc=True) #Activate for additional info if there is any
MIP variables are defined by a MIP and linked to CMOR variables.
They hold information like the variables' unit, etc.