JQ - Merge two arrays

Nam Nguyen picture Nam Nguyen · May 15, 2018 · Viewed 9.7k times · Source

I'm looking for JQ query that allows me to merge 2 arrays variables (not files) and also take me to overwrite first array with newer value from second array. For example:

#!/bin/bash -e

firstArrayVariable='
    [
        {
            "Key": "A B",
            "Value": "1 2"
        },
        {
            "Key": "C D",
            "Value": "3 4"
        },
        {
            "Key": "E F",
            "Value": "5 6"
        },
        {
            "Key": "G H",
            "Value": "9 10"
        }
    ]
'

secondArrayVariable='
    [
        {
            "Key": "A B",
            "Value": "1 2"
        },
        {
            "Key": "C D",
            "Value": "3 4"
        },
        {
            "Key": "G H",
            "Value": "11 12"
        },
        {
            "Key": "J K",
            "Value": "15 16"
        }
    ]
'

jq \
    --compact-output \
    --raw-output \
    --arg jqSecondArrayVariable "${secondArrayVariable}" \
    '. + $jqSecondArrayVariable // empty' \
<<< "${firstArrayVariable}"

I could not get it to work and I got following error

jq: error (at :19): array ([{"Key":"A ...) and string ("\n [\n ...) cannot be added

What I expect the result of the merged array is

[
    {
        "Key": "A B",
        "Value": "1 2"
    },
    {
        "Key": "C D",
        "Value": "3 4"
    },
    {
        "Key": "E F",
        "Value": "5 6"
    },
    {
        "Key": "G H",
        "Value": "11 12"
    },
    {
        "Key": "J K",
        "Value": "15 16"
    }
]

any helps would be very appreciated!

UPDATED

I tried to use --argjson as @peak suggested, it concatenate array but it could not merge 2 arrays. Result I got now is an array with duplicated objects

Answer

RomanPerekhrest picture RomanPerekhrest · May 15, 2018

Assuming that 2 input arrays are named as firstArr and secondArr respectively.

with group_by() (to group objects by the crucial key "Key") and map() functions:

jq --argjson arr1 "$firstArr" --argjson arr2 "$secondArr" -n \
'$arr1 + $arr2 | group_by(.Key) | map(.[-1])'

The output:

[
  {
    "Key": "A B",
    "Value": "1 2"
  },
  {
    "Key": "C D",
    "Value": "3 4"
  },
  {
    "Key": "E F",
    "Value": "5 6"
  },
  {
    "Key": "G H",
    "Value": "11 12"
  },
  {
    "Key": "J K",
    "Value": "15 16"
  }
]

Alternatively and in such particular case, a faster way, you could apply the following trick with unique_by() function:

jq --argjson arr1 "$firstArr" --argjson arr2 "$secondArr" -n '$arr2 + $arr1 | unique_by(.Key)'