Using Base Maps with Non-standard Coordinate Systems in LeafletJS

Since LeafletJS seems to be what all of the cool kids are using these days and it shows no signs of slowing down, I thought that it would be fun to figure out how to use Leaflet to view AGRC's awesome base map services. This presented a unique challenge since they are not in a projection that is supported out-of-the-box by Leaflet (UTM Zone 12 NAD83). However, I found that it is possible with the help of a few additional JavaScript libraries. So, here's the solution:
ESRI-Leaflet & ArcGIS Basemaps
You'll notice that I've loaded these libraries in addition to the latest version of Leaflet:

• Esri-Leaflet Plugin
• Proj4js
• Proj4Leaflet Plugin

The implementation was not that complex once I got all of the numbers right. First I create a new Proj4Leaflet coordinate reference system which I pass into the map constructor. Then I use the Esri-Leaflet Plugin to set up a new TiledMapLayer and add it to the map.
Now you can be one of the cool kids too!

Mac OSX + VMware Fusion + ESRI's ArcGIS Server

While there's endless arguments about whether Mac's or PC's are better for web development, there's not much argument that Mac OSX is the most popular platform for web development today. A few years ago I noticed this trend and decided to make the switch from Windows to Mac for my personal computer. For the most part I have not looked back. It was really nice to be able to follow along with tutorials online and use all of the great tools that are built for the Mac. However, this post is not about convincing you to make the switch; it's about how to use a Mac to develop ArcGIS Server JavaScript applications. It's about how to have your cake and eat it to.

When I finally asked to make the switch to a Mac at work I was faced with a problem. It was not a big deal to spin up a VM with windows server to host ArcGIS Server. However, I didn't want to develop from within my VM. I wanted to continue to use the great development environment that I had in OSX. I also didn't want to write apps that hit ArcGIS Server cross-domain. Hitting ArcGIS Server from localhost on my Mac like this was my goal:

After a few months of messing around, I finally came up with a stable solution that works well. I now do all of my testing and development in OSX and am still able to hit ArcGIS Server with relative urls (/ArcGIS/rest/services...) from within my apps. I accomplished this with a few tricks within the virtual machine software that I use, VMware Fusion.

The first goal was to be able to serve projects that are located within my OSX file system through IIS on my Windows Server 2008 virtual machine. This is accomplished by setting up a shared folder in VMware Fusion (Settings -> Sharing).

Then I set up a virtual directory in IIS that points to that shared folder.

Make sure that you have the \\vmware-host\ in your path. I had to hand type this in to get it to work. This enabled me to hit my web apps that were stored on my Mac from my Windows IIS. However, to hit my VM from my Mac, I still had to use it's ip address. This was a big pain because by default it is not static which meant I was always having to check my VM's ip address and I couldn't bookmark anything.

Finally, I came across an article showing how you can assign a static ip address to your guest and then forward localhost on your host to your guest. The first step is to assign a static ip address to your guest VM by appending /Library/Preferences/VMware Fusion/vmnet8/dhcpd.conf. You can read the article for the specifics of what to add. My addition looks like this:

I don't think that the host name matters as long as you have the correct mac (hardware ethernet) address from your VM.

This change gave my VM the permanent ip address of 192.168.247.100. Then by adding line 55 to /Library/Preferences/VMware Fusion/vmnet8/nat.conf I was able to forward localhost on my Mac to IIS on my VM...

and voila!

I can now write and debug my apps on my Mac but have them served up through IIS on my windows VM.

The ESRI API for JavaScript/Dojo Build System Saga Continues...

That's right, my entire app JS is in this one file *including @ESRI modules* http://t.co/06S4KCtkuR cc @DavidSpriggs @derekswingley
— Scott Davis (@ScottAGRC) May 21, 2013

This is my third post on this topic. First, there was discard layers. Then AMD threw a curve ball. And now a little bird told me that, starting at version 3.4, there is a special build of ESRI's API for JavaScript that is straight up minified AMD modules with no layers files. I'm not sure why this hasn't been more publicized. It's just a little minification away from releasing their source coded. And, as I will outline below, it allows you to build an honest-to-goodness layer file without any of the superfluous code that past versions required. This means that you can build all of the JavaScript code for your application including Dojo, ESRI and your own modules into a single, minified file.

To load the new AMD build of the ESRI API for JavaScript you just append "amd" onto the end of the url. For example, js.arcgis.com/3.6amd (3.5 requires something different because something is messed up with their build: http://js.arcgis.com/3.5amd/js/dojo/dojo/dojo.js). You should be able to just make this one change in your app and it should still work. You may need to explicitly require a few more ESRI modules now that you are not getting a bunch up front with their layer file. With this build you have to load each module individually. So it's going to slow down your load time significantly. It took our boilerplate from 128 requests to 337 requests to load the unbuilt, source version of the app. Needless to say that you should never use this url in production! In fact, I don't even use it during development because it takes so long to load.

The real usefulness in the new AMD build happens when you feed it through the Dojo Build System. The first step is to scrape it to your machine. In order to do this, I needed a list of all of the ESRI modules and other resources like images and .html files. I started with their argument aliases doc and then filled in the gaps by hand. It was easy to see what files I was missing since the build system returns errors for missing dependencies. After I had a good list of files, I used a bash script to pull the files down to my local machine. Here's the script:

The files cannot be fed directly into the build system as is. You'll notice that, in the script above, I had to fix some artifacts from their build (e.g. define('dep1,dep2,dep3'.split(','))...) as well as some paths to the dojo directories. After running this script you have the entire ESRI JavaScript API as separate modules that are ready for the build system. This is so ridiculously close to having the actual source code (just a bit of minification), I wonder if they are just going to release it in the future.

After getting the ESRI modules onto your computer, it's just a matter of pointing to them in your build profile and you are good to go.

You can see the entire set up for this process in AGRC's JavaScript Project Boilerplate. Hopefully the next blog post is about how awesome it is that ESRI has finally released their source code. ;)

ESRI JSAPI 3.4 and the Dojo Build System

[Update: The Saga Continues]

In a previous post, I outlined how I use the Dojo Build System to optimize my web app code for production. Specifically I showed how I get around the problem of working with ESRI's ArcGIS API for JavaScript library which has already been run through the build system. However, with their recent upgrade to AMD-style module loading my handy trick of using:

dojo['require']('esri.map');

... to fool the build system into skipping 'esri...' modules imports didn't work anymore. I had no idea how to exclude modules when loading them like this:

define(['esri.map'], function (Map) { /* ... */ });

So I headed over to #dojo to see if the experts had any ideas. Fortunately for me, brianarn was there and was aware of the problem. After some brain storming, we came up with the idea to use a custom loader plugin for loading ESRI modules. Since the build system doesn't try to flatten modules that are imported with nested requires, we hoped that importing them through the plugin would solve my problem. The plugin was a relatively simple implementation:

You can put this module within any package and use it like this:

define(['app/EsriLoader!esri/map'], function(Map) { /* ... */ });

Using the plugin to load ESRI module effectively prevents the build system from trying to include them in your layer files thus allowing the build script to complete successfully. Of course, none of this hacking would be needed if ESRI would just release their source code. :)

If you find a better way of getting around this problem or have any other suggestions please let me know in the comments section below.

How to Get Started Using Aptana Studio 3 for ArcPy Development (Screencast)

Just put together a short (10 min) video of how to set up Aptana Studio 3 and it's Pydev module for developing python scripts that use ESRI's ArcPy module. Let me know what you think!

Cross-posted on http://gis.utah.gov/developer/.

From Tinkerer to Developer; Or How I Got My Dream Job

Today I got an interesting email from a GIS Analyst in Baltimore

Mr. Davis, 

I am a GIS Specialist who is wrapping up an MS in GIS and looking forward to applying my newly acquired skills to projects here at Baltimore City’s DOT. I recently stumbled onto your blog (http://geospatialscott.blogspot.com/) through an esri forum posting (http://forums.arcgis.com/threads/32892-Google-Streetview-and-Javascript-api) and was quite impressed with the breadth of your programming skills and abilities. 

I was asked if I could try to create a program that would display accident history reports based on intersections in a GIS. My GIS application development experience is limited to what I learned through my coursework and I do not have a programming background. My question for you is how did you go about developing your programming skills as a GIS professional? Do you know of any resources that might aid me in going from a GIS analyst to a creative GIS developer like yourself? I see a heavy need for this type of GIS Developer work here at Baltimore DOT and I would like to contribute as much as possible, utilizing all the skills and resources available to me. 

I’m sure you are super busy, but any suggestions or input would be much appreciated from this aspiring GIS(lowercase)p. 

Thank you in advance, [name withheld]

This caused me to reflect upon the past few years and how I've arrived at such an awesome job. I work for the Utah AGRC as a Geospatial Developer and I love my job. There are very few days that I'm not excited to get to work and create something cool. Every day I'm challenged by new problems to solve through code.

I've worked in my current position for almost two years and am just now becoming comfortable calling myself a developer. This is because I'm entirely self-taught and am naturally self-conscience about my knowledge and skill as a developer. My formal education was in Geography with an emphasis in GIS; no computer science classes at all. I started my career as a GIS Analyst working for several local municipalities.

Here are a few things that I think have contributed to my transformation from a GIS Analyst that had no programming experience to a Geospatial Developer.

30 Minutes Per Day

At the beginning of every GIS job that I have had I have always asked my supervisor if he would allow me to spend 30 minutes per day on learning something new. Universally the response has been, "Really!? You want to better yourself!? Heck yes you can." OK, that may not be the exact wording, but you get the idea. My managers have always been happy to give me that time. I believe that it's because they see it as an investment in their employee.

You Gotta Love it Baby

In order to have the motivation to teach yourself something as complicated and frustrating as programming you have to enjoy it. Find something that excites you and then learn about it. If you are not excited to learn about it, you better move on to something else.

"Give someone a program, you frustrate them for a day; teach them how to program, you frustrate them for a lifetime." - David Leinweber

— Programming Wisdom (@CodeWisdom) June 19, 2012

Learn By Doing Real Work

It's important to me to learn by doing something related to a real project rather than a demo or example project. This helps me stay invested in it as well as lets me know if this technology will really work for my environment. For example, recently I've been reading about backbone.js. Instead of trying to work through an example project that has not relation to anything that I would ever build, I've been reading through the examples and trying to translate them into one of my current projects. By doing this I'm finding that backbone may not be the best solution for my projects. I'm not sure that I would have come to this conclusion as quickly if I had buried myself in demos. Demo's are super useful for showing me how something works, but if I'm writing something, I want it to be connected to my world.

Find a Yoda

Having someone smarter than you to whom you can ask questions is a necessity. I really feel like this is what has made the difference for me. It was only when I was able to get past my own self-conscientiousness and ask smart people questions that I really felt like I made progress. 

Fortunately, in my experience, most programmers are more than happy to give you a little advice and point you in the right direction. I've had several 'famous' JavaScript people respond to my questions on twitter within minutes.

The best ones won't give you the answer right away, but will give you just enough info for you to find the answer on your own. @SteveAGRC is a master at this and has been a great mentor for me. As he would say, "you don't learn anything by keeping your mouth shut."

So I hope that this is a beginning to an answer for my new friend in Baltimore. Maybe a later post will be about specific languages and technologies that I think are the best to learn (JavaScript & Python). If you have any other suggestions for this aspiring developer, please leave a comment.

Phonegap + Leaflet + TileMill = Offline Mobile Maps

Recently I've been researching a mobile project that will require offline-capable base maps. After unsuccessfully finding a solution already built I decided to try wire together Phonegap, Leaflet, and TileMill's .mbtiles. After a few late nights I was able to see it come together and sent out a quick tweet. This has generated quite a few requests for more information so I threw up a repo on GitHub to demo what I was able put together.

The main idea is to download the .mbtiles to the device using Phonegap's File API. Since .mbtiles are just sqlite databases I was able to use a SQLitePlugin to open them up (thanks, coomsie!). I did run into a problem getting this plugin to read BLOB fields from the .mbtiles database. However, after a bit of poking I was able to get it to work.

Once I had access to the encoded image data it was only a matter of writing a custom leaflet tile layer (TileLayer.MBTiles.js) inspired by a similar one that coomsie had done. One of the big secrets was passing {scheme: 'tms'} into the constructor.

Initially the performance was quite good on the iPad 2 and the iPhone 4 that I tried it out on. However, after cleaning up the project in preparation for uploading it to Github, the performance has suffered a bit. Not sure what I did, but I'll post an update when I get it figured out.

This is my first experience with Phonegap and Objective-C so any suggestions for improvements would be greatly appreciated.

[Added on 4-26-12]
P.S. The app downloads the .mbtiles file automatically from my dropbox account the first time that it runs and stores it in the Documents folder locally. Each subsequent time that it runs it uses this local file. So after the initial download you should be able to open up the app and see tiles while in air-plane mode.