Surprises and Gotchas When Working With JSON

This is a list of things about Go's encoding/json package which, over the years, have either confused or surprised me when I first encountered them.

Many of these things are mentioned in the official package documentation if you read it carefully enough, so in theory they shouldn't come as a surprise. But a few of them aren't mentioned in the documentation at all — or at least, they aren't pointed out explicitly — and are worth being aware of!

1. Map entries are sorted alphabetically
2. Byte slices are encoded as base-64 strings
3. Nil and empty slices are encoded differently
4. Integer, time.Time and net.IP values can be used as map keys
5. Angle brackets and ampersands in strings are escaped
6. Trailing zeroes are removed from floats
7. Using omitempty on an zero-valued struct doesn't work
8. Using omitempty on a zero-value time.Time doesn't work
9. There is a 'string' struct tag
10. Non-ASCII punctuation characters aren't supported in struct tags
11. Decoding a JSON number into an interface{} yields a float64
12. Don't use More() to check if there are remaining JSON objects in a stream
13. String values returned by custom MarshalJSON() methods must be quoted

Map entries are sorted alphabetically

When encoding a Go map to JSON, the entries will be sorted alphabetically based on the map key. For example, the following map:

m := map[string]int{
    "z": 123,
    "0": 123,
    "a": 123,
    "_": 123,
}

Will be encoded to the JSON:

{"0":123,"_":123,"a":123,"z":123}

Byte slices are encoded as base-64 strings

Any []byte slices will be converted to a base64-encoded string when encoding them to JSON. The base64 string uses padding and the standard encoding characters, as defined in RFC 4648. For example, the following map:

m := map[string][]byte{
    "foo": []byte("bar baz"),
}

Will be encoded to the JSON:

{"foo":"YmFyIGJheg=="}

Nil and empty slices are encoded differently

Nil slices in Go will be encoded to the null JSON value. In contrast, an empty (but not nil) slice will be encoded as an empty JSON array. For example:

var nilSlice []string
emptySlice := []string{}

m := map[string][]string{
    "nilSlice":   nilSlice,
    "emptySlice": emptySlice,
}

Will be encoded to the JSON:

{"emptySlice":[],"nilSlice":null}

Integer, time.Time and net.IP values can be used as map keys

It's possible to encode a map which has integer values as the map keys. These integers will be automatically converted to strings in the resulting JSON (because the keys in a JSON object must always be strings). For example:

m := map[int]string{
    123: "foo",
    456_000: "bar",
}

Will be encoded to the JSON:

{"123":"foo","456000":"bar"}

In addition, Go allows you to encode maps with keys that implement the encoding.TextMarshaler interface. This means that you can also use time.Time and net.IP values as map keys out-of-the-box. For example:

t1 := time.Now()
t2 := t1.Add(24 * time.Hour)

m := map[time.Time]string{
    t1: "foo",
    t2: "bar",
}

Will be encoded to the JSON:

{"2009-11-10T23:00:00Z":"foo","2009-11-11T23:00:00Z":"bar"}

Note that trying to encode a map with any other type of key will result in a json.UnsupportedTypeError error.

Angle brackets and ampersands in strings are escaped

If a string contains angle brackets<> these will be escaped to \u003c and \u003e in the JSON output. Likewise the & character will be escaped to \u0026. This is to prevent some web browsers from accidentally interpreting the JSON as HTML. For example:

s := []string{
    "<foo>",
    "bar & baz",
}

Will be encoded to the JSON:

["\u003cfoo\u003e","bar \u0026 baz"]

If you need to prevent these characters being escaped, you should use a json.Encoder instance and call SetEscapeHTML(false). An example is here.

Trailing zeroes are removed from floats

When encoding a floating-point number with a fractional part that ends in zero(es), any trailing zeroes will not appear in the JSON. For example:

s := []float64{
    123.0,
    456.100,
    789.990,
}

Will be encoded to the JSON:

[123,456.1,789.99]

Using omitempty on an zero-valued struct doesn't work

The omitempty directive never considers a struct type to be empty — even if all the struct fields have their zero value, and you use omitempty on those fields too. It will always appear as an object in the encoded JSON. For example:

s := struct {
    Foo struct {
        Bar string `json:",omitempty"`
    } `json:",omitempty"`
}{}

Will be encoded to the JSON:

{"Foo":{}}

There’s a long-standing proposal which discusses changing this behavior, but the Go 1 compatibility promise means that it's unlikely to happen any time soon. Instead, you can get around this by making the field a pointer to a struct, which works because omitempty considers nil pointers to be empty. For example:

s := struct {
    Foo *struct {
        Bar string `json:",omitempty"`
    } `json:",omitempty"`
}{}

Using omitempty on a zero-value time.Time doesn't work

Using omitempty on a zero-value time.Time field won't hide it in the encoded JSON. This is because the time.Time type is a struct behind the scenes and, as mentioned above, omitempty never considers a struct type to be empty. Instead, the string "0001-01-01T00:00:00Z" will appear in the JSON (which is the value returned by calling the MarshalJSON() method on an zero-value time.Time. For example:

s := struct {
    Foo time.Time `json:",omitempty"`
}{}

Will be encoded to the JSON:

{"Foo":"0001-01-01T00:00:00Z"}

There is a 'string' struct tag

Go provides a string struct tag directive which forces the data in an individual field to be encoded as a string in the resulting JSON. For example, if you want to force an integer to be represented as a string instead of an JSON number you can use the string directive like so:

s := struct {
    Foo int `json:",string"`
}{
    Foo: 123,
}

And this will be encoded to the JSON:

{"Foo":"123"}

Note that the string struct tag directive will only work on fields which contain float, integer or bool types. For any other type it will have no effect.

Non-ASCII punctuation characters aren't supported in struct tags

When using struct tags to change key names in JSON, any tags containing non-ASCII punctuation characters will be ignored. Notably this means that you can't use en or em dashes, or most currency signs, in struct tags. For example:

s := struct {
    CostUSD string `json:"cost $"` // OK
    CostEUR string `json:"cost €"` // Contains the non-ASCII punctuation character €. Will be ignored.
}{
    CostUSD: "100.00",
    CostEUR: "100.00",
}

Will be encoded to the following JSON (notice that the struct tag renaming the CostEUR field has been ignored):

{"cost $":"100.00","CostEUR":"100.00"}

Likewise, any struct tags containing non-ASCII punctuation characters will be ignored when decoding values from a JSON object into a struct, and the struct field will be left with its zero value. For example the following code:

js := []byte(`{"cost $":"100.00","cost €":"100.00"}`)

s := struct {
    CostUSD string `json:"cost $"`
    CostEUR string `json:"cost €"`
}{}

err := json.Unmarshal(js, &s)
if err != nil {
    log.Fatal(err)
}

fmt.Printf("%+v", s)

Will print out:

{CostUSD:100.00 CostEUR:}

This can be annoying in situations where you need to decode a JSON object that has keys containing non-ASCII characters, and you can't change the JSON. To work around this limitation, you can decode to a map as an intermediary step, and then copy the data from the map to the struct. For example the following code:

js := []byte(`{"cost $":"100.00","cost €":"100.00"}`)

var aux map[string]string

err := json.Unmarshal([]byte(js), &aux)
if err != nil {
    log.Fatal(err)
}

s := struct {
    CostUSD string `json:"cost $"`
    CostEUR string `json:"cost €"`
}{
    CostUSD: aux["cost $"],
    CostEUR: aux["cost €"],
}

fmt.Printf("%+v", s)

Will print out:

{CostUSD:100.00 CostEUR:100.00}

Decoding a JSON number into an interface{} yields a float64

When decoding a JSON number into an interface{}, the value will have the underlying type float64 — even if it is an integer in the original JSON.

If you want to get the value as an integer (instead of a float64) the most robust approach is to decode the JSON using a json.Decoder instance with the UseNumber() method set on it. This will decode all JSON numbers to the underlying type json.Number instead of float64, and you can then access the number as an integer using its Int64() method. For example:

js := `{"foo": 123, "bar": true}`

var m map[string]interface{}

dec := json.NewDecoder(strings.NewReader(js))
dec.UseNumber()

err := dec.Decode(&m)
if err != nil {
    log.Fatal(err)
}

i, err := m["foo"].(json.Number).Int64()
if err != nil {
    log.Fatal(err)
}

fmt.Printf("foo: %d", i)

Will print:

foo: 123

Don't use More() to check if there are remaining JSON objects in a stream

When processing a stream of JSON objects with json.Decoder, don't use the More() method to check if there is a remaining object in the stream. Depsite its name, More() is not designed for this purpose^†, and trying to use it in this way may cause some subtle problems.

^†The More() method is intended to be used in conjunction with Token(), and exists specifically to check if there is another element in the array or object currently being parsed.

For example, if you use it when decoding an invalid JSON stream like {"name": "alice"}{"name": "bob"}] (notice the additional square bracket at the end) it won't result in an error (when it should!). Like so:

js := `{"name": "alice"}{"name": "bob"}]`

dec := json.NewDecoder(strings.NewReader(js))
for {
    var user map[string]string

    err := dec.Decode(&user)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("%v\n", user)

    // Don't do this!
    if !dec.More() {
        break
    }
}

This code will run without error and output:

map[name:alice]
map[name:bob]

The correct technique to see if a stream contains another JSON object is to check for an io.EOF error, which will be returned when there are no more objects to process in the stream. Like so:

js := `{"name": "alice"}{"name": "bob"}]`

dec := json.NewDecoder(strings.NewReader(js))
for {
    var user map[string]string

    err := dec.Decode(&user)
    if err != nil {
        if errors.Is(err, io.EOF) {
            break
        }
        log.Fatal(err)
    }

    fmt.Printf("%v\n", user)
}

Running this will correctly result in an error, as we would expect given the invalid input:

map[name:alice]
map[name:bob]
2009/11/10 23:00:00 invalid character ']' looking for beginning of value

String values returned by custom MarshalJSON() methods must be quoted

If you are creating a custom MarshalJSON() method which returns a string value, you must wrap the string in double quotes before returning it, otherwise it won't be interpreted as a JSON string and will result in a runtime error. For example:

type Age int

func (age Age) MarshalJSON() ([]byte, error) {
    encodedAge := fmt.Sprintf("%d years", age)
    encodedAge = strconv.Quote(encodedAge) // Wrap the string in quotes before returning.
    return []byte(encodedAge), nil
}

func main() {
    users := map[string]Age{
        "alice": 21,
        "bob":   84,
    }

    js, err := json.Marshal(users)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("%s", js)
}

Will result in the following JSON being printed:

{"alice":"21 years","bob":"84 years"}

If, in the code above, you didn't quote the return value from MarshalJSON() you will get the error:

2009/11/10 23:00:00 json: error calling MarshalJSON for type main.Age: invalid character 'y' after top-level value