Python Script To Update Current Stream Gauge Data In New AGRC Flood Map

We use a python script to scrape data from the USGS and NWS web sites to update our data in the SGID. It runs every two hours through Windows Scheduled Tasks. The script’s workflow is as follows:

First it loops through all of the features in our stream gauges feature class (SGID93.WATER.StreamGaugesNHD).

For each feature, it uses a USGS id (SourceFeature_ID) to build a url to hit their Instantaneous Values web service. 

# get json object



data = json.loads(urllib2.urlopen(r'http://waterservices.usgs.gov/nwis/iv?format=json&site=' + id).read())

This is an example of one of the urls: http://waterservices.usgs.gov/nwis/iv?format=json&site=09413700. It then uses the json library to parse the data and get the values that we are interested in. These values are used to populate the appropriate fields in our feature class.

def getJsonValue(variableCode, data):
    for ts in data['value']['timeSeries']:
        if ts['variable']['valueType'] == variableCode:
            value = ts['values'][0]['value'][0]['value']
            return value

The NOAA data is served up via an rss feed which means xml. The minidom object from the xml.dom library came in handy here for parsing the xml data.

# get noaa data
gaugeID = row.getValue('GuageID')
if gaugeID:
    ndata = minidom.parse(urllib2.urlopen('http://water.weather.gov/ahps2/rss/fcst/' + gaugeID.lower() + '.rss'))
    descriptionText = ndata.getElementsByTagName('description')[2].firstChild.nodeValue
    descriptionList = descriptionText.split('<br />')
    row.setValue('HIGHEST_FORECAST', descriptionList[5].split()[2].strip())
    row.setValue('HIGHEST_FORECAST_DATE', getNOAADate(descriptionList[6].split('Time:')[1].strip()))
    row.setValue('LAST_FORECAST', descriptionList[8].split()[2].strip())
    row.setValue('LAST_FORECAST_DATE', getNOAADate(descriptionList[9].split('Time:')[1].strip()))

So in the end we have one feature class that combines real-time data from multiple sources. You can check out a copy of the script here.

ArcPy.Mapping Module Makes Complex PDF Creation Easy

Recently, I was presented with a problem that was a perfect opportunity for trying out ESRI's ArcPy Python site package. We have a series of map documents that are set up with Data Driven Pages to export various maps for each of the counties of Utah. Each mxd had a different theme. The goal was to develop a script that would export all of the Data Driven Pages for each mxd and then combine them by county. After a few hours of work I had 72 lines of code that did just that. Here's what I came up with:

I used only two modules for this script: arcpy.mapping and os (great for working with the file system).

# import modules
import arcpy.mapping, os

# variables
baseFolder = os.getcwd() # current working directory
outputFolder = baseFolder + r'\PDFs'

The os module was great for deleting the old files and getting a list of the map documents.

# clear out old pdfs
print '\nDeleting old PDFs...'
oldPDFs = os.listdir(outputFolder)
for f in oldPDFs:
    os.remove(outputFolder + '\\' + f)

# get list of all files in the folder
print '\nGetting list of mxds...'
allItems = os.listdir(baseFolder)
# filter out just .mxd's
mxdFileNames = [(x) for x in allItems if x.endswith('.mxd')]
mxdFileNames.sort()

The DataDrivenPages class was the key class in the ArcPy.Mapping module for this script. It is obtained through the MapDocument class. Here I start to loop through the mxd's and get a reference to the DataDrivenPages object that I am interested in.

# loop through mxds
for name in mxdFileNames:
    print '\nProcessing: ' + name
    
    # get mxd
    mxd = arcpy.mapping.MapDocument(baseFolder + '\\' + name)
    
    # get datadrivenpages object
    ddp = mxd.dataDrivenPages

Once I've got the DataDrivenPages object, then I start to loop through all of the pages.

    # loop through pages
    pg = 1
    while pg <= ddp.pageCount:
        # change current page
        ddp.currentPageID = pg
        
        # get name of current page
        name = ddp.pageRow.getValue('NAME')
        print name

Before I export the page, I check to see if there is already an existing pdf for that particular county. If there is I export the page to a temp PDF file and then use the PDFDocument::appendPages() function to add it to the existing PDF. If not, then I just export it out to a new PDF.

        # check to see if there is already a pdf file created for this county
        pdfFile = outputFolder + '\\' + name + '.pdf' 
        if os.path.exists(pdfFile):
            print 'Existing pdf found. Appending...'
            
            # open PDF document
            pdf = arcpy.mapping.PDFDocumentOpen(pdfFile)
            
            # output to temporary file
            tempFile = outputFolder + '\\temp.pdf'
            ddp.exportToPDF(tempFile, 'CURRENT')
            
            # append to existing file
            pdf.appendPages(tempFile)
            
            # delete temp file
            os.remove(tempFile)
            
            # clean up variables
            del pdf
            
        else: # file does not exist, export to new file
            print 'No existing pdf found. Exporting to new pdf.'
            ddp.exportToPDF(pdfFile, 'CURRENT')
        
        # increment page number
        pg = pg + 1

Then, all that's left if a little clean up.

    # clean up variables
    del mxd, ddp
    
raw_input('Done. Press any key to exit...')

And that's it! Here's the entire script and an example output pdf.

Great Blog Post on Custom Dojo Dijits

In his new blog "Enterprise Dojo", Dan Lee has an outstanding article titled, "Introduction to Custom Dojo Widgets". It gives a great first taste of developing your own custom Dojo Dijits. I was really excited to learn about the slick way to reference DOM nodes within your dijit using dojoattachpoint. Take 10 minutes and give it a read!

Vehicle Tracking With The JavaScript API

Recently, we have been investigating alternatives to our current AVL solution. The problems with our current solution are licensing costs (big $$ per workstation) and ease of use. I thought that I'd give it a try with ESRI's JavaScript API and this is what I came up with.

The vehicles are graphics that are refreshed from the server every 5 seconds using the JavaScript setTimeout() function. I was surprised at how quickly the browser could clear the graphics and reload them from the server. It looks like they are animated instead of reloaded. Here's a sample of the code that I used:

function AddVehicleGraphics(){
    SANDY.qt.execute(SANDY.q, function(fSet){
        function sortGraphics(a,b){
            return (a.attributes.fleetID < b.attributes.fleetID) ? -1 : 1; 
        }
        
        // record bread crumbs, if any
        dojo.forEach(SANDY.crumbsVehicles, function(key){
            var crumbGraphic = dojo.filter(SANDY.avlGraphicsLayer.graphics, function(item){
                return item.attributes.mapKey == key;
            })[0];
            SANDY.avlGraphicsLayer.remove(crumbGraphic); // required to allow appropriate refresh on history graphics layer
            crumbGraphic.setInfoTemplate(SANDY.crumbsTipTemplate);
            crumbGraphic.setSymbol(SANDY.crumbSymbol);
            
            // get associate graphics layer
            var gLayer = SANDY.mapDij.map.getLayer(key);
            gLayer.add(crumbGraphic);
        });
        
        // clear graphics layer
        SANDY.avlGraphicsLayer.clear();
        SANDY.chartsDij.clearContent();
        
        // sort the graphics alphabetically so they are sorted when adding to active vehicles table.
        fSet.features.sort(sortGraphics);
        dojo.forEach(fSet.features, function(g){
            var symbol;
            switch (g.attributes.defaultIcon) {
                case "PLOW TRUCK":
                    symbol = dojo.clone(SANDY.truckSymbol);
                    break;
                case "SWEEPER":
                    symbol = dojo.clone(SANDY.sweeperSymbol);
                    break;
                case "SUV":
                    symbol = dojo.clone(SANDY.puSymbol);
                    break;
            }
            
            // rotate symbol
            symbol.setAngle(g.attributes.headingDegrees);
            g.setSymbol(symbol);
            
            // round mph
            g.attributes.speedMPH = Math.round(g.attributes.speedMPH);
            
            // add to graphics layer
            SANDY.avlGraphicsLayer.add(g);
            
            // add to vehicles list / bold all vehicles that have updated in the last 30 minutes
            var content;
            var activeCutOff = new Date().setMinutes(new Date().getMinutes() - 30);
            if (new Date(g.attributes.timeAtCoordinate) > activeCutOff){
                content = "<b>" + g.attributes.fleetID + "</b>";
            } else {
                content = g.attributes.fleetID;
            }
            SANDY.chartsDij._addContent(content);
            
            // update date updated text
            var now = new Date();
            dojo.byId("update-date").innerHTML = now.toLocaleTimeString();
            dojo.style("update-date-background", "backgroundColor", "white");
        });
        
        if (fSet.features.length === 0){
            SANDY.chartsDij._addContent(SANDY.noVehiclesMsg);
        }
    }, function(error){
        // change background to indicate error
        dojo.style("update-date-background", "backgroundColor", "red");
    });
}

The other cool thing that I was able to do in this project was link to live UDOT traffic cameras. After getting permission from them, I was able to determine the unique URL for each of them from the CommuterLink website and build a feature class with their locations and urls. It was easy after that to publish them as a map service and load them as graphics onto my map. I used a custom infoTip window from ESRI's code gallery and some mouse events to show the camera images.

// camera graphics
dojo.connect(SANDY.mapDij.map.graphics, "onMouseOver", function(evt){
    // store old symbol to save for mouse out function
    SANDY.origSymbol = evt.graphic.symbol;
    
    // get orig color rgb's
    var r = SANDY.origSymbol.color.r;
    var g = SANDY.origSymbol.color.g;
    var b = SANDY.origSymbol.color.b;
    
    // highlight opacity
    var a = 0.5;
    
    // change symbol to highlight symbol
    var biggerSize = evt.graphic.symbol.size + 4;
    evt.graphic.setSymbol(dojo.clone(evt.graphic.symbol).setSize(biggerSize).setColor([r, g, b, a]));
    
    SANDY.iTip.setContent(evt.graphic.getContent());
    SANDY.iTip.show(evt.screenPoint, SANDY.mapDij.map.height, SANDY.mapDij.map.width);
});
dojo.connect(SANDY.mapDij.map.graphics, "onMouseOut", function(evt){
    // change symbol back
    evt.graphic.setSymbol(SANDY.origSymbol);
    
    SANDY.iTip.hide();
});
dojo.connect(SANDY.mapDij.map.graphics, "onClick", function(evt){
    // hide infowindow
    SANDY.mapDij.map.infoWindow.hide();
});

The next step after we upgrade to ArcGIS Server 10 is to use the new TimeSlider dijit to enable users to view historical data. Sadly, because I'm changing jobs, I will not be around to implement this cool functionality.  :(

Any other suggestions? Anyone else using the JavaScript API for vehicle tracking? I'd love to hear your experiences.

Onward and Upward - New Job

Recently I was offered a new position at the State of Utah's AGRC. I'm way excited to be working with such an amazing group of people and can't wait to get started. I'll miss all of the cool people and projects that I'll be leaving behind here at Sandy. But the good news is that I'll be doing a lot more programming. Hopefully I'll still have the time and motivation to keep up the blog. Even though I'm pretty sure that the only people that read it are me and my wife...OK...truthfully, I have to read it to my wife to get her to read it. ;)

If you are interested in filling my position at Sandy, you can go here for the job announcement.

I start next week. Wish me luck!

Cool Census Data App From ESRI Applications Prototype Lab

Here's a cool Silverlight app from the ESRI folks that like to live on the cutting edge. I really like the idea of using the physics engine to reposition the graphics. Obviously the app needs refining (what's with the tool bar on the left that looks disabled until you hover?), but the concept is cool. You can check out their related blog post here, or go right to the app to try it out yourself.

Modify Feature With Sketch

This was something that I really missed when I upgraded to 10. To install go to Customize -> Customize Mode -> Add from file... and select the ModifyWithSketch.tlb file in the folder ModifyWithSketch\bin\Debug. Then you will have a new tool in the Commands tab under "Developer Samples" called "Continue sketching an existing..". You can drag this to any toolbar. To use, just select an existing feature and then click on the tool.