Using the Foursquare API to get a list of trending venues

In this tutorial we will explore how to consume JSON web APIs using Haskell. In particular, we will use the Google Geocoding API to convert an address into a lat long, and then plug that information into the Foursquare API to get a list of trending venues. Topics covered include:

• Making HTTP[S] requests.
• Parsing JSON using Data.Aeson.
• Using type classes to make it easy to add support for new endpoints.
• Propagating errors using the IO monad.

This tutorial is aimed at intermediate Haskell programmers.

Basics: Making an HTTP Request

At its core, consuming a web API is a matter of making HTTP requests to some server with the desired parameters (see: REST). We'll be using the Network.HTTP.Conduit library for making these requests.

The interface is simple:

simpleHttp :: MonadIO m => String -> m ByteString

MonadIO is just a type class for monads that can embed IO operations -- including IO itself. The first parameter to this function is a URI, and it returns a ByteString -- which is a just a buffer.

Here's an example:

import Network.HTTP.Conduit
main = do -- The GitHub API is convenient as it doesn't require authentication
          response <- simpleHttp "https://api.github.com/repos/yesodweb/yesod" 
          print response

You can expect to see output along the lines of:

Chunk "{\"id\":237904,\"name\":\"yesod\",\"full_name\":\"yesodweb/yesod\",
\"owner\":{\"login\":\"yesodweb\",\"id\":930379,\"avatar_url\":\"https://sec
ure.gravatar.com/avatar/c224026a2005e5ce9a0e1a6defb9f893?d=https://a248.e
.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-org-420.png\",
\"gravatar_id\":\"c224026a2005e5ce9a0e1a6defb9f893\",\...

It's not very pretty, but this JSON response contains a lot of juicy information about the Yesod repository, including its URL, description, and owner. Take a look for yourself.

Extracting Structured Data: Parsing with Aeson

Data.Aeson provides tons of handy utilities for dealing with JSON. For our purposes, we mostly care about decode -- which takes a ByteString and returns a kind of JSON AST.

import Data.Aeson
import Data.ByteString.Lazy.Char8(pack)
main = print $ (decode (pack "{\"key\": \"val\"}") :: Maybe Value)

So, what's a Value?

data Value = Object !Object -- The exclamation mark (!) is a strictness annotation.
           | Array !Array
           | String !Text
           | Number !Number
           | Bool !Bool
           | Null

As you can see, it corresponds very nicely to the structure of a JSON document. Using simpleHTTP and decode together, you should be able to send a request to an endpoint and parse it into a JSON AST. However, in the next section we'll take a step back and start building some utilities to help us build out different endpoints in a generic manner.

Building Out a Framework For Endpoints

What is an endpoint (besides a miserable pile of secrets)? Since we're dealing with GET requests only, for our purposes it's basically a URI with some query parameters. We would like to encapsulate those parameters in a typed data structure, and provide a way to build a URI using those parameters. This notion is captured by the following type class:

class Endpoint a where
  buildURI :: a -> String

For an example, let's build out the geocoding endpoint. As a quick reminder, geocoding is the process of converting an address like "New York City" into a latitude and longitude. We'll need a lat/long for our subsequent call to the Foursquare API.

From the Google Geocoding API Documentation, we can see that the form of the geocoding request is fairly simple: http://maps.googleapis.com/maps/api/geocode/output?parameters, where the two required parameters (that we care about) are:

• address: The address that you want to geocode.
• sensor: Boolean indicating whether the request came from a device with a location sensor. For our program this will always be false.

We encapsulate these parameters with the following ADT:

data GeocoderEndpoint =
  GeocodeEndpoint { address :: String, sensor :: Bool }

To convert this data structure into a URI, we need to prepend the API endpoint path to a string consisting of the query parameterized fields.

instance Endpoint GeocoderEndpoint where
  buildURI GeocodeEndpoint { address = address, sensor = sensor } =
    let params = [("address", Just address), ("sensor", Just $ map toLower $ show sensor)]
    in "http://maps.googleapis.com/maps/api/geocode/json" ++ renderQuery True params

As you can see, its fairly simple. I've glossed over the implementation of renderQuery :: Bool -> [(String, Maybe String)] -> String -- it converts a set of parameters into a string like "?key1=value1&key2=value2" (making sure to URL-encode the values). To see the full definition, check out the source on GitHub.

Detour: Aeson Parsers

Before we get to decoding the response from the geocoder endpoint, let's take a detour and explore a powerful feature of Aeson: Parsers.

Parsers are closely tied to the FromJSON type class we saw earlier. From the Aeson documentation, a FromJSON is "A type that can be converted from JSON, with the possibility of failure." Its only method is parseJSON :: Value -> Parser a.

A Parser is a monad that encapsulates that possibility of failure, as well as a sequence of operations which convert a Value (JSON AST) into some type a. A naive implementation of parseJSON could simply inspect the Value and return an a based on that information, but Aeson also provides a few handy combinators operating within the Parser Monad. We'll be using .:, which allows you to easily access Object fields. For example:

{-# LANGUAGE OverloadedStrings #-}
import Data.Aeson
import Data.Aeson.Types
import Data.Text(Text)
import Data.ByteString.Lazy.Char8

data Venue = Venue { venueId :: Text, name :: Text } deriving Show

instance FromJSON Venue where
  parseJSON val = do let Object obj = val -- Use pattern matching to extract an Object
                     venueId <- obj .: "id"
                     name <- obj .: "name"
                     return $ Venue venueId name

main = print venue
  where json = "{\"id\": \"some venue id\", \"name\": \"bob's venue\"}"
        venue = decode json >>= parseMaybe parseJSON :: Maybe Venue

Now that we have parser basics down, we should be able to model the geocoder result as an ADT, and parse it from a decoded Value.

Modeling the Geocoder Result

Now that we can construct a URI for calling the geocoder, we need to make sense of the response. I'd suggest hitting the endpoint in a web browser to get a rough idea of what the geocoder returns.

The response is fairly generic, with support for multiple results and lots of extra metadata. The only thing we care about is the lat/long of the first result. In JavaScript notation: response.results[0].geometry.location.

Since this structure is fairly nested, I built a helper method called navigateJson :: Value -> [Text] -> Parser Value. This takes a Value and a list of field names and walks down the tree -- for example, if you provided {a: {b: 2}} and ['a', 'b'] as parameters, it would return 2. I'll elide the definition here; check out the full source on GitHub.

With navigateJSON, building the Parser should be trivial. Access the 'results' field of the response; get the first entry; navigate thru ['geometry', 'location']; return the 'lat' and 'lng' parts of that object.

Before we continue, let's define an ADT to encapsulate this type of result.

type LatLng = (Double, Double)
data GeocodeResponse = GeocodeResponse LatLng
  deriving Show -- Handy for debugging

Now we can write our parser by declaring an instance of the FromJSON type class.

instance FromJSON GeocodeResponse where
  parseJSON (Object obj) =
    do (Array results) <- obj .: "results"
       (Object location) <- navigateJson (results V.! 0) ["geometry", "location"]
       (Number lat) <- location .: "lat"
       (Number lng) <- location .: "lng"
       -- Use realToFrac to convert from Aeson Numbers into simple Doubles.
       return $ GeocodeResponse (realToFrac lat, realToFrac lng)

Putting It All Together (Part 1)

So far we have a method to build URIs for endpoints and a way to parse the JSON response from those endpoints into a structure we can use. Since these steps are encapsulated by the generic type classes FromJSON and Endpoint, its easy to build a general function to call any endpoint. Let's start with a type signature: callJsonEndpoint :: (FromJSON j, Endpoint e) => e -> IO j. "Take an endpoint (with data about the call), make some HTTP request, and parse the response into some JSON object." This process must take place in the IO monad, since we're making a network request.

We'll use simpleHTTP like earlier. To run our parser, we'll use a variant of decode called eitherDecode -- using the error message from Either, we can provide more helpful errors (via IO's fail).

callJsonEndpoint :: (FromJSON j, Endpoint e) => e -> IO j
callJsonEndpoint e =
  do responseBody <- simpleHttp (buildURI e)
     case eitherDecode responseBody of
       Left err -> fail err
       Right res -> return res

With callJsonEndpoint, we could call the geocoder endpoint like so:

(GeocodeResponse latLng) <- callJsonEndpoint $ GeocodeEndpoint "568 Broadway, New York, NY" False

Hooking Up Foursquare

Now that we have a nifty little framework, we can come back to the motivation for this tutorial: retreiving a list of trending venues around an address. Foursquare makes this easy with an endpoint called "/venues/trending". The parameter we care about is the lat/long ("ll"), but you can optionally provide a limit on the results and a radius for your search area.

data FoursquareEndpoint =
    VenuesTrendingEndpoint { ll :: LatLng, limit :: Maybe Int, radius :: Maybe Double }

instance Endpoint FoursquareEndpoint where
  buildURI VenuesTrendingEndpoint {ll = ll, limit = limit, radius = radius} =
    let params = [("ll", Just $ renderLatLng ll), ("limit", fmap show limit), ("radius", fmap show radius)]
    in "https://api.foursquare.com/v2/venues/trending" ++ renderQuery True params

The docs tell us that this returns a list of "venues", which is another JSON object. For our simple example, we choose to care about two fields: the venue ID, and its name.

data Venue = Venue { venueId :: String, name :: String } deriving Show

The whole response from this endpoint (list of venues) is encompassed by the following structure:

data VenuesTrendingResponse = VenuesTrendingResponse { venues :: [Venue] } deriving Show

Implementing FromJSON instances for these structures is left as an exercise for the reader (with the source on GitHub available for the lazy ;).

Foursquare Authorization

Even though we've implemented data structures, parsing logic, and a URI builder for the venues/trending endpoint, there remains one restriction upon using the Foursquare API: you must have access credentials.

The venues/trending endpoint is "userless", so we don't need to go through OAuth, but we still need an API key/secret. I've covered the process of obtaining these credentials in an appendix below, so let's assume we have those for the remainder of this tutorial.

To start, let's define a structure to represent the needed credentials:

data FoursquareCredentials = FoursquareCredentials { clientId :: String, clientSecret :: String }

We can actually build upon the Endpoint framework to build a type of "authorized Foursquare endpoint" that wraps another endpoint and appends the necessary access credentials. Let's call this an AuthorizedFoursquareEndpoint:

data AuthorizedFoursquareEndpoint = AuthorizedFoursquareEndpoint FoursquareCredentials FoursquareEndpoint

In order to build an instance of Endpoint for AuthorizedFoursquareEndpoint, all we have to do is build the URI of the inner endpoint, and then append "client_id" and "client_secret" parameters.

instance Endpoint AuthorizedFoursquareEndpoint where
  buildURI (AuthorizedFoursquareEndpoint creds e) = appendParams originalUri authorizationParams
    where originalUri = buildURI e
          authorizationParams = [("client_id", Just $ clientId creds),
                                 ("client_secret", Just $ clientSecret creds),
                                 ("v", Just foursquareApiVersion)]

Finally, let's define a handy helper that lets us authorize an endpoint using an infix syntax (e.g. endpoint `authorizeWith` creds):

authorizeWith = flip AuthorizedFoursquareEndpoint

Putting It All Together (Part 2)

To recap: we've built a system for describing web API endpoints; we learned how to use Aeson's FromJSON to parse the results of those endpoints; and we implemented a function to call those endpoints using that functionality.

At the beginning of this tutorial, we wanted to retrieve a list of trending venues about an address. At this point, we can achieve that goal.

1. Call Google's geocoder endpoint. Store the lat/long.
2. Plug that lat/long into Foursquare's trending venues endpoint.
3. Display those trending venues to the user.

For our Main.hs, we'll also use getLine to retrieve access credentials. Here goes:

targetAddress = "568 Broadway, New York, NY"
main :: IO ()
main =
  do putStrLn "API key?"
     apiKey <- getLine
     putStrLn "API secret?"
     apiSecret <- getLine
     let creds = FoursquareCredentials apiKey apiSecret

     (GeocodeResponse latLng) <- callJsonEndpoint $ GeocodeEndpoint targetAddress False
     let venuesTrendingEndpoint = VenuesTrendingEndpoint latLng Nothing Nothing `authorizeWith` creds
     (VenuesTrendingResponse venues) <- callJsonEndpoint venuesTrendingEndpoint
     let printVenue v = putStrLn $ "- " ++ name v
     mapM_ printVenue venues

And that's it! Once again, see the full source on GitHub for a complete implementation of the techniques described in this tutorial.

Appendix: Obtaining Foursquare API Credentials

1. Sign up for Foursquare if you don't already have an account.
2. Log in.
3. Go to My Apps (this is linked to from the developer website).
4. Click "Create A New App".
5. Enter some details. The only 3 required fields here are the app name, the download URL, and the redirect URI. Since we're not using OAuth, the redirect URI doesn't matter. I would recommend adding any path on a domain you own for these two URIs.
6. Once you click "Save Changes", you'll be directed to the details of your app. This includes your client key and secret, which you can use to run this example!