GRACE-Basic App Tutorial
The NASA GRACE mission involves two satellites launched in 2002 that orbit the earth on a pattern that covers the globe. One satellite follows the other on a precise trajectory and the distance between the satellites is meausured with a laser. As the satellites pass over mountains and other anomolies in the earth's mass, the distance between the satellites changes very slightly. These changes are used to map the earth's gravitational field - more specifically, changes in the gravitational field. Changes in the gravitational field are generally caused by changes in water storage: snow, ice, surface water, soil moisture, and groundwater. Thus, the GRACE mission allows us to track water storage trends across the globe.
In this exercise, we will build a simple app that displays a time series representing the change in water volume for a selected region defined by a shapefile. This relies on a Python script called SHBAAM developed by Cedric David of the NASA Jet Propulsion Laboratory. The script takes the following as input:
- GRACE data file for entire world for entire program history in a netCDF file
- Scaling factors file
- Shapefile for region of interest
and produces the following:
- Clipped netCDF file with gridded and scaled GRACE data for region of interest
- CSV file with time series of GRACE anomalies averaged over region of interest
We will create an app that lets the user pick from one of three regions and we will display the region on a map and the time series CSV file in a HighCharts plot. We will also display the shapefile on a map.
This app is called GRACE-Basic to distinguish it from the more sophisticated GRACE Tethys app hosted on the BYU Tethys server.
Getting Started
Before completing this tutorial, you should have done the following:
- Install Tethys and all dependencies
- Install GeoServer and make sure it is working properly.
If you followed the standard install procedure, including the dockers, you should be ready to go.
While not necessary, you may also wish to complete the Spatial Dataset Services and GeoServer tutorial before completing this tutorial since will be using GeoServer.
Note: This tutorial relies on Python 3 packages. If your default environment is not Python 3, be sure to start your Python 3 environment before proceeding. See the Conda chapter for more details.
Also, if necessary, open a terminal window and do the following:
1. Start tethys virtual environment.
$ t
2. If you are using dockers, start them as follows:
$ tethys docker start
Or if you only want to start GeoServer, you can do this:
$ tethys docker start -c geoserver
If you are using a standalone version of GeoServer, make sure the server is running.
3. You may also need to start the Tethys database.
$ tstartdb
Create Spatial Dataset Service Setting
Before continuing, we need to configure a spatial dataset service in the Tethys admin portal. If you have already done this as part of the Spatial Dataset Services and GeoServer tutorial, you can skip this and go to the next section.
1. To get to the admin page you will need to start Tethys.
$ tms
2. Browse to http://http://localhost:8000/.
3. Select the dropdown menu next to your username in the top right-hand corner of the screen and select the Site Admin link.
4. Select the Spatial Dataset Services link from the Tethys Services section and then press the Add Spatial Dataset Service button.
6. Create a new Spatial Dataset Service named "default" of type GeoServer, and then enter both the endpoint and public endpoint.
If you are using docker, enter: http://localhost:8181/geoserver/rest/
If you are using a default installation of GeoServer, enter: http://localhost:8180/geoserver/rest/
7. Using the URL shown in the terminal, and fill out the username and password. The default username and password for GeoServer is "admin" and "geoserver", respectfully. You do not need to enter an API key.
You may also wish to explore the GeoServer web admin interface by visiting : http://localhost:8181/geoserver/web/ (or use http://localhost:8180/geoserver/web/ for default installations). Login using “admin” and “geoserver”.
Scaffold a New App
Next we will create our app.
1. Quit tethys manage by selecting Ctrl-C
2. Path into the tethysdev folder.
3. Type the following to create a new app:
$ tethys scaffold grace_basic
4. Accept the defaults at all the prompts.
5. Path into the folder you just created. Note the name.
$ cd tethysapp-grace_basic
6. Then run the setup script:
$ python setup.py develop
7. Restart tethys
$ tms
8. Launch your app and check it out.
Copy the GRACE Files
Before continuing, we need to copy some files that we will use in the app. We will copy the files into the app_workspace folder. This where you put files that need to be accessed by your app. There is also a folder called user_workspaces where you would store files uploaded by users.
1. Click on the following link and download a zip archive containing the files we will be using:
2. Unzip the archive.
After uncompressing the archive, you should see two subfolders:
output
shapefiles
You may wish to explore the contents of these folders.
3. Copy the entire contents of both of these folders to the the following subfolder:
/tethysapp-grace_basic/tethysapp/grace_basic/workspaces/app_workspace/
When you are done, your files/directories should look like this:
Take a minute to explore the contents of these two folders. The shapefiles folder contains three shapfiles:
nepal
reg12_texas
reg18_calif
These shapefiles were imported to the SHBAAM script to produce a set of CSV files that contain the total change in water volume vs time for each of the three regions. These CSV files are stored in the output folder. We will display these as time series plots in our app.
Changing the App Content
Now we are ready to start modifying the code. We will start with some basics.
1. Open your favorite python editor and browse the file structure of the app
2. Open home.html in the templates/grace_basic folder.
3. Change the app_content block as follows (copy and paste this code block):
{% block app_content %}
<h1>Welcome</h1>
<p>This app illustrates how to perform basic Tethys functions using GRACE groundwater data of groundwater volume changes over time since 2002.</p>
<p>Click on a region using the links on the left to plot GRACE data for a selected region.</p>
{% endblock %}
4. Refresh/relaunch your app to see the changes.
App Navigation Items
Next we will change the app navigation items.
1. In the home.html file, add the following just above the app_content section:
{% block app_navigation_items %}
<li class="title">Regions</li>
<li>
<a href="{% url 'grace_basic:home' %}">Instructions</a>
</li>
<li>
<a href="{% url 'grace_basic:home' %}">Nepal</a>
</li>
<li>
<a href="{% url 'grace_basic:home' %}">Region 12 (Texas)</a>
</li>
<li>
<a href="{% url 'grace_basic:home' %}">Region 18 (California)</a>
</li>
{% endblock %}
2. Save and reload. Notice the changes to the navigation bar on the left.
Adding a Time Series Plot
Next we will change our app to create a time series plot for the region selected. This will use the CSV files in the output folder that we copied to the app_workspace directory. Do the following:
1. Open the controller.py file.
2. Copy and paste the following to the top of the file:
from tethys_sdk.gizmos import * import csv, os from datetime import datetime
3. Then copy-paste the following function and append it at the bottom:
@login_required
def home_graph(request, id):
"""
Controller for home page to display a graph and map.
"""
# Set up the graph options
project_directory = os.path.dirname(__file__)
app_workspace = os.path.join(project_directory, 'workspaces', 'app_workspace')
csv_file = os.path.join(app_workspace, 'output' , id, 'hydrograph.csv')
with open(csv_file, 'rb') as f:
reader = csv.reader(f)
csvlist = list(reader)
volume_time_series = []
formatter_string = "%m/%d/%Y"
for item in csvlist:
mydate = datetime.strptime(item[0], formatter_string)
volume_time_series.append([mydate, float(item[1])])
# Configure the time series Plot View
grace_plot = TimeSeries(
engine='highcharts',
title=id + ' GRACE Data',
y_axis_title='Volume',
y_axis_units='cm',
y_min=None,
series=[
{
'name': 'Change in Volume',
'color': '#0066ff',
'data': volume_time_series,
},
],
width='100%',
height='300px'
)
context = {'grace_plot': grace_plot,
'reg_id': id}
return render(request, 'grace_basic/home.html', context)
NOTE: If you are using a Tethys 1.x installation, you may need to remove the "y_min=None," line. See note at the end of this tutorial on the time series bug.
This function loads a special version of the home page that contains a time series plot for the selected region. One of the arguments to the function is id, which identifies the region of interest ("nepal" vs. "reg12_texas", etc.). The first part of the function creates a path to the CSV file associated with the id parameter in the app_workspace folder using the os.path.join function from the os Python library. This function takes a series of folder names and builds a file path that has the proper delimeters ("\" vs."/") based on the operating system being used. It then uses some functions from the cvs Python library to read the contents of the CSV file. The datetime values from the CSV file are then converted to the correct format and the datetime and volume pairs are appended to a Python list. Finally, the list is used to configure a plot view gizmo with the proper formatting and the plot and id parameters are added to the context and passed to the renderer for final step of building the HTML page to be rendered.
Next we need to add a URL map to the app.py file.
4. Open the app.py file
5. Change the URL maps to the following:
url_maps = (
UrlMap(
name='home',
url='grace-basic',
controller='grace_basic.controllers.home'),
UrlMap(name='home_graph',
url='grace-basic/home/{id}',
controller='grace_basic.controllers.home_graph')
)
This creates a URL map for the home_graph version of the home page and indicates that the region id is passed as a URL variable. It then ensures that the proper controller is called to prepare the content for the page.
Finally, we need to update the navigation items and add the plot view gizmo to home.html.
6. Open home.html and copy the following to the second line:
{% load tethys_gizmos %}
7. Copy the following to the app_navigation_items block:
{% block app_navigation_items %}
<li class="title">Regions</li>
<li{% if not reg_id %} class="active"{% endif %}>
<a href="{% url 'grace_basic:home' %}">Instructions</a>
</li>
<li{% if reg_id == 'Nepal' %} class="active"{% endif %}>
<a href="{% url 'grace_basic:home_graph' id='nepal' %}">Nepal</a>
</li>
<li{% if reg_id == 'Reg12' %} class="active"{% endif %}>
<a href="{% url 'grace_basic:home_graph' id='reg12_texas' %}">Region 12 (Texas)</a>
</li>
<li{% if reg_id == 'Reg18' %} class="active"{% endif %}>
<a href="{% url 'grace_basic:home_graph' id='reg18_calif' %}">Region 18 (California)</a>
</li>
{% endblock %}
Note the we have changed the navigation items such that when the user clicks on the Instructions link, it loads the home page using the original home() controller function, but if any of the region links are clicked, it calls the home_graph() controller function and passes the id of the associated region.
8. Finally, copy the following to the app_content block:
{% block app_content %}
{% if not reg_id %}
<h1>Welcome</h1>
<p>This app illustrates how to perform basic Tethys functions using GRACE groundwater data of groundwater volume changes over time since 2002.</p>
<p>Click on a region using the links on the left to plot GRACE data for a selected region.</p>
{% else %}
{% gizmo plot_view grace_plot %}
{% endif %}
{% endblock %}
This block now uses a Python IF statement to check to see if a region id is defined. If not, it loads the original form of the home page with the instructions. But if the region id is defined, it loads the plot view gizmo with the time series for the selected region.
8. Save and reload your app to test it.
Adding a Map
Finally, we will add a map just above the time series plot displaying the shapefile for the selected region. To do this, the shapfiles must first be uploaded to GeoServer. But this only needs to be done once. To handle this, we will add some code that runs when the main home page (the version with the instructions) is loaded. This code will check to see if the region shapefiles are loaded into GeoServer. If not, it will upload them from the shapefiles folder in the app_workspace.
1. Open controllers.py and add the following to the top of the file:
import random
import string
2. Copy-paste the following code to the top of the file, just below all of the import statements and above the home() function:
WORKSPACE = 'grace_basic_app'
GEOSERVER_URI = 'http://www.example.com/grace-basic-app'
def check_shapefile(id):
'''
Check to see if shapefile is on geoserver. If not, upload it.
'''
geoserver_engine = get_spatial_dataset_engine(name='default')
response = geoserver_engine.get_layer(id, debug=True)
if response['success'] == False:
#Shapefile was not found. Upload it from app workspace
#Create the workspace if it does not already exist
response = geoserver_engine.list_workspaces()
if response['success']:
workspaces = response['result']
if WORKSPACE not in workspaces:
geoserver_engine.create_workspace(workspace_id=WORKSPACE, uri=GEOSERVER_URI)
#Create a string with the path to the zip archive
project_directory = os.path.dirname(__file__)
app_workspace = os.path.join(project_directory, 'workspaces', 'app_workspace')
zip_archive = os.path.join(app_workspace, 'shapefiles', id, id + '.zip')
# Upload shapefile to the workspaces
store = ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(6))
store_id = WORKSPACE + ':' + store
geoserver_engine.create_shapefile_resource(
store_id=store_id,
shapefile_zip=zip_archive,
overwrite=True
)
This code defines a function called check_shapefile that takes the id (name) of a region as an argument and checks to see if there is already a shapefile with that name loaded on GeoServer. If not, it uploaded the associated shapefile from the files stored in the app_workspace\shapefiles folder.
Next, we will modify the home() function to call the check_shapefile() function, once for each region.
3. Modify the home() function as follows:
@login_required()
def home(request):
"""
Controller for the app home page.
"""
'''
Check each region to ensure that the shapefile for the region
is stored in geoserver.
'''
check_shapefile('nepal')
check_shapefile('reg18_calif')
check_shapefile('reg12_texas')
context = {}
return render(request, 'grace_basic/home.html', context)
Now we are ready to add the code to display the shapefile for a region on map.
4. Add the following to the top of the controller.py file:
from tethys_sdk.services import get_spatial_dataset_engine import urlparse
5. In the same file, add the following to the home_graph() function, just below the first statement (and comments) and above the code that sets up the plot view:
# Set up the map options
'''
First query geoserver to get the layer corresponding to id, then parse
one of the URLs in the response DICT to get the bounding box.
Then parse the box to get lat/long to properly center the map
'''
geoserver_engine = get_spatial_dataset_engine(name='default')
response = geoserver_engine.get_layer(id, debug=False)
kmlurl = response['result']['wms']['kml']
parsedkml = urlparse.urlparse(kmlurl)
bbox = urlparse.parse_qs(parsedkml.query)['bbox'][0]
bboxitems = bbox.split(",")
box_left = float(bboxitems[0])
box_right = float(bboxitems[2])
box_top = float(bboxitems[3])
box_bottom = float(bboxitems[1])
centerlat = (box_left + box_right) / 2
centerlong = (box_top + box_bottom) / 2
map_layers = []
geoserver_layer = MVLayer(
source='ImageWMS',
options={'url': 'http://localhost:8181/geoserver/wms',
'params': {'LAYERS': id},
'serverType': 'geoserver'},
legend_title=id,
legend_extent=[box_left, box_bottom, box_right, box_top],
legend_classes=[
MVLegendClass('polygon', 'Boundary', fill='#999999'),
])
map_layers.append(geoserver_layer)
view_options = MVView(
projection='EPSG:4326',
center=[centerlat, centerlong],
zoom=4,
maxZoom=18,
minZoom=2,
)
map_options = MapView(height='300px',
width='100%',
layers=map_layers,
legend=True,
view=view_options
)
This codes queries geoserver to figure out the bounding box of the shapefile to properly orient the map. Then it initializes the map view gizmo to display the layer on geoserver associated with the selected region.
3. Next, update the context at the bottom of the function to look like this:
context = {'map_options': map_options,
'grace_plot': grace_plot,
'reg_id': id}
4. Finally, open the home.html file and add the map view gizmo to the app_content block by adding the following line:
{% gizmo map_view map_options %}
inside the else statement and just above the line for the plot view gizmo. After pasting it in, the block should look like this:
{% block app_content %}
{% if not reg_id %}
<h1>Welcome</h1>
<p>This app illustrates how to perform basic Tethys functions using GRACE groundwater data of groundwater volume changes over time since 2002.</p>
<p>Click on a region using the links on the left to plot GRACE data for a selected region.</p>
{% else %}
{% gizmo map_view map_options %}
{% gizmo plot_view grace_plot %}
{% endif %}
{% endblock %}
This simply places the map above the time series plot when the home page is rendered using the home_graph() controller.
5. Save and reload.
We are done! Your app should be fully functional at this point.
Truncated Plot Error
If you are using a Tethys 1.x Installation and your time series plots are only showing the top half, you need to do the following:
1. Path to the following location in your text editor:
~/tethys/src/tethys_gizmos/gizmo_options/
2. Open the plot_view.py file.
3. Scroll down to the TimeSeries class and location the following code in the initiator method:
4. Comment out the line that sets the min to zero as follows:
5. Save and reload the app.
This is a bug/limitation that has been fixed in Tethys 2.0.
Solution
You can access a solution to this tutorial from the following repository:
https://github.com/BYU-Hydroinformatics/tethysapp-grace_basic
See the GitHub unit for information on how to clone a Tethys app using a GitHub repository.
