NWM Forecast App Tutorial

The new National Water Model (NWM) provides continuous hydrologic forecasting for the nearly 2.7 million reaches that make up the National Hydrography Dataset (NHD Plus). As part of the Young Innovators Program at the National Water Center over the past three years several applications to access and use the output have been created using Tethys. The NWM Viewer application can be used to select individual stream reaches and view the short, medium, long range and analysis and assimilation model configurations, while other applications use the API functions to obtain a forecast and compute statistics, compare with other gage data and forecasts, and generate derivative products such as floodplain maps. In this exercise, we will build a simple app that accesses the NWM Viewer API and displays the forecast for the short and medium range configurations for the reach that coincides with Two Mile Creek as it enters the Black Warrior River near the University of Alabama Campus in Tuscaloosa.

The script takes the following as input:

  • The Reach ID (COMID) of the Two Mile Creek as it enters the Black Warrior River
  • The NWM Forecast configuration (In this example the Medium and Short Range are demonstrated, but it could be extended to other configurations and or times).

and accessing the API of the NWM Viewer produces the following:

  • A plot of the current forecast

We will create an app that lets the user pick from one of these configurations and then display the current forecast. We will also demonstrate adding the NHD maps as a Webbing Mapping Service.

Solution Source Code

Source code of this example is stored at:

https://github.com/BYU-Hydroinformatics/tethysapp-nwm_example

which has the following branches:

“master”: the completed app once you finish the last section “Adding a View for the Short Range Forecast”
“step_1_3”: Scaffold a New App Section-- Step 1-3
“step_6_7_8”: Scaffold a New App Section -- Step 6-8
“medium_range_step_1_6”: Adding a View for the Medium Range Forecast Step Section 1-6
“short_range_step_1_5”: Adding a View for the Short Range Forecast Section Step 1-5

If you get stuck on any particular section, you copy/download a solution from any of these branches using the steps outlined in the GitHub unit.

Getting Started

Before completing this tutorial, you should have done the following:

  • Install Tethys 2.0 (or newer)
  • Install a Python Code Editor (e.g. PyCharm or Spyder), or any text editor

Also, if necessary, open a terminal window and do the following:

1. Start tethys virtual environment. 

$ t

2. Start Tethys

(tethys) $ tstartdb
(tethys) $ tms

3. Launch your browser and view Tethys Portal http://127.0.0.1:8000 or http://localhost:8000.

4. Sign in with default admin credentials: admin/pass

Scaffold a New App

First we will create a new empty app. For each of the following console commands, enter only the code following the "(tethys) $".

Step 1: Create a new empty app

If you do not already have a development folder create one by pathing to your home directory and typing:

(tethys) $ mkdir ~/tethysdev

Make your tethysdev your current directory

(tethys) $ cd ~/tethysdev

Create a new app using the scaffold command. You will be prompted to enter metadata about your app such as, proper name, version, author, and description. All of these metadata are optional. You can accept the default value by pressing Enter, repeatedly.

(tethys) $ tethys scaffold nwm_example

Step 2: Install app on Tethys Portal

Change to the new app directory and run setup.py.

(tethys) $ cd ~/tethysdev/tethysapp-nwm_example

Install this new app to Tethys:

(tethys) $ python setup.py develop

Step 3: View the new app in Tethys Portal

Start/Restart Tethys:

(tethys) $ tstartdb
(tethys) $ tms

Go to the browser at http://127.0.0.1:8000/apps/

You should see this:

The components of this page come from two different templates using the Django paradigm:

Step 4: Clean up base.html

Navigate to the folder

tethysdev/tethysapp-nwm_example/tethysapp/nwm_example/templates/nwm_example

and open the base.html file in a text editor.

Remove:

 <li class="title">App Navigation</li>
 <li class="active"><a href="">Home</a></li>
 <li><a href="">Jobs</a></li>
 <li><a href="">Results</a></li>
 <li class="title">Steps</li>
 <li><a href="">1. The First Step</a></li>
 <li><a href="">2. The Second Step</a></li>
 <li><a href="">3. The Third Step</a></li>
 <li class="separator"></li>
 <li><a href="">Get Started</a></li>

From block:

{% block app_navigation_items %}
  …
{% endblock %}

Restart Tethys and View your app:

Step 5: Clean up home.html

Navigate to the folder:

tethysdev/tethysapp-nwm_example/tethysapp/nwm_example/templates/nwm_example

and open the home.html file in a text editor.

Remove this code:

  <h1>Welcome to your Tethys App!</h1>
  <p>Take advantage of beautiful typography to organize the content of your app:</p>
  <h1>Heading 1</h1>
  <h2>Heading 2</h2>
  <h3>Heading 3</h3>
  <h4>Heading 4</h4>
  <h5>Heading 5</h5>
  <h6>Heading 6</h6>

From block:

{% block app_content %}
  …
{% endblock %}

Remove this code:

  {% gizmo save_button %}
  {% gizmo edit_button %}
  {% gizmo remove_button %}
  {% gizmo previous_button %}
  {% gizmo next_button %}

From block:

{% block app_actions %}
  …
{% endblock %}

Restart Tethys and View your app:


Note: If you encounter errors connecting to the database, use the following command: 

(tethys) $ tstartdb

Add a Map and Web Mapping Service

Next we will add a map to our app using the Map View gizmo and populate it using a web mapping service.

Step 1: Add Home button to base.html

Open the base.html file in a text editor and INSERT this code:

 {% url 'nwm_example:home' as home_url %}
  <li class="title">Navigation</li>
  <li class="{% if request.path == home_url %}active{% endif %}"><a href="{{ home_url }}">Home</a></li>

Into this block:

{% block app_navigation_items %}
  …
{% endblock %}

Step 2: Chage home.html to add a Map

Navigate to the folder

tethysdev/tethysapp-nwm_example/tethysapp/nwm_example/templates/nwm_example

and open the home.html file in a text editor. Replace the contents with the following:

Insert this code:

{% gizmo nwm_example_map %}

Into this block:

{% block app_content %}
  ...
{% endblock %}

Step 3: Add a base map and the Watershed WMS Service

Navigate to the:

tethysdev/tethysapp-nwm_example/tethysapp/nwm_example

folder and open controllers.py. Replace the contents with the following code:

import datetime as dt
from django.shortcuts import render
from django.contrib.auth.decorators import login_required
from django.shortcuts import reverse
from tethys_sdk.gizmos import MapView, Button, MVLayer, MVLegendClass
import plotly.graph_objs as go
from tethys_sdk.gizmos import PlotlyView
#from .helpers import get_nwm_forecast

@login_required()
def home(request):
    """
    Controller for the app home page.
    """
    # Tiled ArcGIS REST Layer
    arc_gis_layer2 = MVLayer(
        source='TileArcGISRest',
        options={'url': 'https://services.nationalmap.gov/arcgis/rest/services/wbd/MapServer'},
        legend_title='Watershed Boundaries',
        legend_extent=[-173, 17, -65, 72]
    )
    nwm_example_map = MapView(
        height='100%',
        width='100%',
        layers=[arc_gis_layer2],
        basemap='OpenStreetMap',
        legend = True
    )
	  
    context = {
        'nwm_example_map': nwm_example_map
    }
	  
    return render(request, 'nwm_example/home.html', context)

Step 4: Restart Tethys and view your app

Start the Tethys Server

(tethys) $ tms

If you encounter errors connecting to the database, use the following command: 

(tethys) $ tstartdb

Your app should look like this:

Adding a View for the Medium Range Forecast

The objective of this next exercise is to create a second page for our app that displays a forecast for the NWM medium range configuration for Two Mile Creek in Tuscaloosa. The forecast will be accessed using the helper function previously created to call the API from the NWM Viewer. Our start date is July 20 and the forecast will last about 10 days.

Step 1: Create a helper function to access the NWM data for a reach

In the same folder you are working in, create a new Python file called helpers.py and add the following:

# Use the API from the NWM Viewer app to get the WaterML text
import requests
def get_nwm_forecast(config, comid, startdate, enddate, forecasttime):
    url = 'https://apps.hydroshare.org/apps/nwm-forecasts/api/GetWaterML/?config=' + config + '&geom=channel_rt&variable=streamflow&COMID=' + comid + '&lon=-87.5658033081755&lat=33.2279708144365&startDate='+startdate+'&endDate='+enddate+'&time='+forecasttime+'&lag=t00z'
    res = requests.get(url).content
    return res

Open controller.py to enable the above Helper function:

Replace:

# from .helpers import get_nwm_forecast

With:

from .helpers import get_nwm_forecast

Step 2: Add a link to the new medium range forecast page

Open the base.html file and edit the block app navigation items. This will create a new button to navigate to the Medium Range page.

Inside this block:

{% block app_navigation_items %}
  ...
{% endblock %}

Directly after this code:

{% url 'nwm_example:home' as home_url %}

Insert this code:

{% url 'nwm_example:medium_range' as medium_range_url %}

Then, after this code:

<li class="{% if request.path == home_url %}active{% endif %}"><a href="{{ home_url }}">Home</a></li>

Insert this code:

<li class="{% if request.path == medium_range_url %}active{% endif %}"><a href="{{ medium_range_url }}">Medium Range Forecast</a></li>

Step 3: Create a new page for the medium range forecast

Create a new html file called medium_range.html by copying another html file and renaming it. Replace the contents with the following code:

{% extends "nwm_example/base.html" %}
{% load tethys_gizmos %}
{% block app_content %}
  <h1>National Water Model</h1>
  <h2>Medium Range Forecast</h2>
  {% gizmo nwm_plot %}
{% endblock %}

Step 4: Add code to request the forecast

Open the controllers.py file and edit the home(request) by editing the context.

Create a new medium_range(request) function by inserting the following code below the home(request) in the controllers.py file. The river “comid” being referenced is Two Mile Creek in Tuscaloosa. This comid could be replaced with any valid comid that is part of the NHD Plus dataset.

@login_required()
def medium_range(request):
    """
    Controller for the Medium Range page.
    """
    dateraw = []
    date1 = []
    value1 = []
    comid = '18228725'
    config = 'medium_range'
    startdate = '2017-07-20'
    enddate = '2017-07-21' #not needed for medium range
    forecasttime = '00'
    watermlstring = str(get_nwm_forecast(config, comid, startdate, enddate, forecasttime))
    waterml = watermlstring.split('dateTimeUTC="')
    waterml.pop(0)
    for e in waterml:
        parser = e.split('"  methodCode="1"  sourceCode="1"  qualityControlLevelCode="1" >')
        dateraw.append(parser[0])
        value1.append(parser[1].split('<')[0])
    for e in dateraw:
        date1.append(dt.datetime.strptime(e, "%Y-%m-%dT%H:%M:%S"))
    nwm_plot = PlotlyView([go.Scatter(x=date1, y=value1)])
    
    context = {    
        'nwm_plot': nwm_plot
    }
    
    return render(request, 'nwm_example/medium_range.html', context)

Step 5: Update the URL Maps

Open the app.py file and make the following edits:

After this code:

        url_maps = (
            UrlMap(
                name='home',
                url='nwm-example',
                controller='nwm_example.controllers.home'
            ),

Insert this code:

            UrlMap(
                name='medium_range',
                url='nwm-example/range/medium',
                controller='nwm_example.controllers.medium_range'
            ),

When finished, the complete app.py should be as follows:

from tethys_sdk.base import TethysAppBase, url_map_maker

class NwmExample(TethysAppBase):
    """
    Tethys app class for Nwm Example.
    """

    name = 'Nwm Example'
    index = 'nwm_example:home'
    icon = 'nwm_example/images/icon.gif'
    package = 'nwm_example'
    root_url = 'nwm-example'
    color = '#8e44ad'
    description = 'Place a brief description of your app here.'
    tags = ''
    enable_feedback = False
    feedback_emails = []

    def url_maps(self):
        """
        Add controllers
        """
        UrlMap = url_map_maker(self.root_url)

        url_maps = (
            UrlMap(
                name='home',
                url='nwm-example',
                controller='nwm_example.controllers.home'
            ),
            UrlMap(
                name='medium_range',
                url='nwm-example/range/medium',
                controller='nwm_example.controllers.medium_range'
            ),
        )

        return url_maps

Step 6: View the changes to your app.

Start the Tethys Server

(tethys) $ tms

If you encounter errors connecting to the database, use the following command:

(tethys) $ tstartdb

Open a web browser and go to the site http://127.0.0.1:8000/apps to view your app.

Adding a View for the Short Range Forecast

Now we will add a second request to access information from the National Water Model to generate a short range forecast.

Step 1: Add link to new short term forecast page

Once again, we will make a link to a new page by making the following edits in base.html:

After this code:

  {% url 'nwm_example:medium_range' as medium_range_url %}

Insert:

  {% url 'nwm_example:short_range' as short_range_url %}

After this code:

  <li class="{% if request.path == medium_range_url %}active{% endif %}"><a href="{{ medium_range_url }}">Medium Range Forecast</a></li>

Insert:

  <li class="{% if request.path == short_range_url %}active{% endif %}"><a href="{{ short_range_url }}">Short Range Forecast</a></li>

Step 2: Create a new View page for the short range forecast

Make a copy of medium_range.html and rename it short_range.html. Replace the contents of the file with the following:

{% extends "nwm_example/base.html" %}
{% load tethys_gizmos %}

{% block app_content %}
  <h1>National Water Model</h1>
  <h2>Short Range Forecast</h2>
  {% gizmo nwm_plot %}
{% endblock %}

Step 3: Add controller for short range forecast

Add the controller code for the short range forecast in controller.py by making the following edits:

Add this new function to the end of this file:

@login_required()
def short_range(request):
    """
    Controller for the Short Range page.
    """

    dateraw = []
    date1 = []
    value1 = []
    comid = '18228725'
    config = 'short_range'
    startdate = '2017-07-20'
    enddate = '2017-07-21'
    forecasttime = '12'
    watermlstring = str(get_nwm_forecast(config, comid, startdate, enddate, forecasttime))
    waterml = watermlstring.split('dateTimeUTC="')
    waterml.pop(0)
    for e in waterml:
        parser = e.split('"  methodCode="1"  sourceCode="1"  qualityControlLevelCode="1" >')
        dateraw.append(parser[0])
        value1.append(parser[1].split('<')[0])

    for e in dateraw:
        date1.append(dt.datetime.strptime(e, "%Y-%m-%dT%H:%M:%S"))

    nwm_plot = PlotlyView([go.Scatter(x=date1, y=value1)])

    context = {
        'nwm_plot': nwm_plot
    }

    return render(request, 'nwm_example/short_range.html', context)

Step 4: Update the URL Maps in app.py

Open the app.py file. After this code:

            UrlMap(
                name='medium_range',
                url='nwm-example/range/medium',
                controller='nwm_example.controllers.medium_range'
            ),

Insert:

            UrlMap(
                name='short_range',
                url='nwm-example/range/short',
                controller='nwm_example.controllers.short_range'
            ),

Step 5: Restart Tethys and View your app

Start the Tethys Server:

(tethys) $ tms

If you encounter errors connecting to the database, use the following command: 

(tethys) $ tstartdb

Open a web browser and go to the site http://127.0.0.1:8000/apps to view your app.

At this point, your app is complete!

 

BYU Hydroinformatics Group

Brigham Young University
Dept. of Civil and Environmental Engineering