Parsing JSON in crystal

I’ve recently been diving more into Crystal Lang, and while it recently reached version 1.0, I still find that examples on how to do certain things are lacking online, here’s my small contribution to the topic of JSON parsing which specifically tripped me up every time.

Recently, I needed to parse JSON data from an API, and I realised there are a couple of different ways to do this with Crystal.

Dynamic JSON parsing in Crystal

We can parse JSON dynamically on the fly, as follows:

require "json"

json_array = JSON.parse("[1, 2, 3]")
json_array[0]              # => 1

While this works, and is fairly simple to understand, I feel it doesn’t make use of Crystal’s type checking, and we may run into unclear runtime errors.

This also requires casting from time to time, as stated in the crystal docs, which might not be ideal.

Crystal JSON parsing into objects

Another approach, which is the one I’ve been trying to favour so far, is to build Crystal objects from JSON, thanks to the JSON::Serializable module which can be included in any class.

While it may seem a little daunting at first, I think this is a very nice way to describe how certain JSON is supposed to be parsed, and avoids writing a lot of object[0]['property_1']['nested_array'][1].

How does it work?

a simple example:

Let’s say we have an object representing a job offer in our data model. Here’s how we can write how to parse JSON containing such data.

class JobOffer
  include JSON::Serializable

  property position : String
end

Here, the JobOffer class will define an attribute called position, of type String, and the key of this information in the JSON payload will be the same as the attribute name.

Here’s some simple JSON that will parsed with this code:

{
  "position": "Senior web developer",
}

Expand with nested data

Now, let’s say that our Job offer has a nested object company, we can simply indicate this with the following:

class Company
  include JSON::Serializable

  property name : String
  property location : String?
end

class JobOffer
  include JSON::Serializable

  property position : String
  property company : Company
end

When parsing a job offer, our JobOffer class will delegate the parsing of the company property to the Company class.

The company has a name, and an optional location (denoted by the question mark).

Here’s some json that could be parsed with our code so far

{
  "position": "Senior web developer",
  "company": {
    "name": "BlueGreen",
    "location": "Anywhere"
  }
}

Parsing arrays of string, with different property name

Say now, that our JSON data looks like this:

{
  "position": "Senior web developer",
  "tags": ["ruby", "crystal", "react"],
  "company": {
    "name": "BlueGreen",
    "location": "Anywhere"
  }
}

On our JobOffer models, we have a property named categories. It would be nice to be able to map our array of tags into a similar data structure in Crystal, and to be able to map it to our internal naming.

Well, there’s actually something made for this use case. We can use an annotation to describe additional behaviour on the field. Coming from ruby, this feels like a nice magic addition to describe behaviour. Here, the key indicates that we want to map the tags section of the json data structure, into our categories array.

The type of our categories property being Array(String) , our tags can be mapped without any additional effort on our side.

class JobOffer
  include JSON::Serializable

  property position : String
  property company : Company

  @[JSON::Field(key: "tags")]
  property categories : Array(String)
end

Massaging the data before conversion

Now, imagine a new property in our json data, called salary, and containing entries such as CAD 12000 , € 7000.

{
  "position": "Senior web developer",
  "tags": ["ruby", "crystal", "react"],
  "salary": "CAD 12000",
  "company": {
    "name": "BlueGreen",
    "location": "Anywhere"
  }
}

On our side, the data model has a property holding salary information. We store the salary amount in a property called salary.

Parsing this data will require a little of processing on the data itself, and this is something that’s quite easy to do with the JSON::Serializable module. We can specify a converter, that will handle the transformation of the data into the formats we’d like.

Here’s how we can achieve it:

class JobOffer
  include JSON::Serializable

  property position : String
  property company : Company

  @[JSON::Field(key: "tags")]
  property categories : Array(String)

  @[JSON::Field(key: "salary", converter: JobOffer::SalaryConverter)]
  property salary : Int32

  class SalaryConverter
    def self.from_json(value : JSON::PullParser) : Int32
      matched_data = match_data(value)

      return 0 unless matched_data

      matched_data["amount"].to_i
    end

    def self.match_data(value)
      /(?<currency>\w+) (?<amount>\d+)/.match(value.read_string)
    end
  end
end

Here, our SalaryConverter class will parse the string provided into the salary property of our json file, and parse the amount out of it.

Going further

Let’s keep our last example, and imagine that now, the data model has two properties holding salary information. We store the salary in a property called salary and the currency in a property called salary_currency.

Essentially, we’d like to map our salary property to 2 different attributes of our JobOffer object. At first, I thought we could specify different converters, for the same JSON property, such as the following:

class JobOffer
  include JSON::Serializable

  property position : String
  property company : Company

  @[JSON::Field(key: "tags")]
  property categories : Array(String)

  @[JSON::Field(key: "salary", converter: JobOffer::SalaryConverter)]
  property salary : Int32

  @[JSON::Field(key: "salary", converter: JobOffer::SalaryCurrencyConverter)]
  property salary_currency : String
end

Unfortunately, it seems that it is currently not possible to have attributes based off of the same JSON property, judging the error given at compilation:

❯ crystal spec job_offer_spec.cr
There was a problem expanding macro 'macro_4707212240'

Code in /usr/local/Cellar/crystal/1.0.0_2/src/json/serialization.cr:159:7

 159 | {% begin %}
       ^
Called macro defined in /usr/local/Cellar/crystal/1.0.0_2/src/json/serialization.cr:159:7

 159 | {% begin %}

Which expanded to:

 > 132 |               end
 > 133 |
 > 134 |             when "salary"
                          ^
Error: duplicate when "salary" in case

Conclusion

JSON parsing in Crystal is actually not that complex, but it took me a few iterations before grasping how this works.

The code presented in the examples throughout this article is available at:

Hopefully this can help some of you building cool things in Crystal. Drop me a tweet at clement_f, I’d be curious to hear out what you’re building 👀.

References

The official Crystal lang docs