FQ(1)

NAME
fq - tool, language and decoders for working with binary formats

SYNOPSIS

fq [OPTIONS] [--] [EXPRESSION] [FILE...]

DESCRIPTION

fq is a tool, language, and set of decoders for working with binary formats and data. In most cases it behaves and feels similar to and it also uses the same expression language. To get the most out of fq it’s recommended to learn more about jq.

It features a structural hex viewer, nested format decoding, slicing and concatenating binary data, bit-level decoding and an interactive REPL with auto-completion.

    # Evaluate "d" (display) for file.mp4
    $ fq d file.mp4
    # structural hex output
    # Evaluate "da" (display all) for file.mp4
    $ fq da file.mp4
    # structural hex output
    # JSON for file.mp4
    $ fq -V . file.mp4
    # JSON output
    # Evaluate ".boxes[0].type" for all *.mp4 files
    # -Vr to output value without quotes
    $ fq -Vr '.boxes[0].type' *.mp4
    ftyp
    # Evaluate "1+2" without reading any input
    $ fq -n 1+2
    3

For more usage examples see the examples section at the end of the documentation.

OPTIONS

--arg NAME VALUE
    Set $NAME to string VALUE

--argdecode NAME PATH
    Set $NAME to decode of PATH

--argjson NAME JSON
    Set $NAME to JSON

--args
    Consume remaining arguments as positional strings

--color-output, -C
    Force color output

--compact-output, -c
    Use compact output

--decode, -d NAME
    Decode format or group (probe)

--from-file, -f PATH
    Read EXPRESSION from file

--help, -h [TOPIC]
    Show help for TOPIC (ex: -h formats, -h mp4)

--include-path, -L PATH
    Add PATH to include search paths

--join-output, -j
    No newline after each output

--jsonargs
    Consume remaining arguments as positional JSON

--monochrome-output, -M
    Force monochrome output

--null-input, -n
    Null input (use input and inputs to read)

--option, -o NAME=VALUE/@PATH
    Set option (ex: -o color=true, see --help options)

    -o addrbase=number
        Number base for addresses

    -o array_truncate=number
        Array display length to truncate

    -o bits_format=string
        Raw bits representation

        -o bits_format=base64
            Base64 string.

        -o bits_format=byte_array
            Array of bytes (zero bit padded if size is not byte aligned).

        -o bits_format=hex
            Hex string.

        -o bits_format=md5
            MD5 hex string (zero bit padded).

        -o bits_format=snippet
            Truncated Base64 string prefixed with bit length.

        -o bits_format=string
            String with raw bytes (zero bit padded if size is not byte aligned). The string is binary safe internally in fq but bytes not representable as UTF-8 will be lost if turned into JSON (default).

        -o bits_format=truncate
            Truncated string.

    -o byte_colors=ranges=string,...
        Byte value colorization

    -o color=true|false
        Use color

    -o colors=key=value,...
        Color scheme

    -o compact=true|false
        Use compact JSON

    -o completion_timeout=number
        Seconds to wait for completion results

    -o depth=number
        Display tree depth limit

    -o display_bytes=number
        Display bytes limit

    -o force=true|false
        Force decode

    -o join_string=string
        String used to join outputs

    -o line_bytes=number
        Number of bytes per display line

    -o raw_string=true|false
        Raw string output

    -o sizebase=number
        Number base for sizes

    -o skip_gaps=true|false
        Skip gaps when representing decode value (arrays) as JSON

    -o string_truncate=number
        String display length truncate

    -o unicode=true|false
        Use unicode

    -o verbose=true|false
        Verbose display

    -o width=number
        Terminal width

--raw-file NAME PATH
    Set $NAME to string content of file

--raw-input, -R
    Read raw input strings (don’t decode)

--raw-output, -r
    Raw string output (without quotes)

--raw-output0
    NUL (zero) byte after each output

--repl, -i
    Interactive REPL

--slurp, -s
    Slurp all inputs into an array or string (-Rs)

--unicode-output, -U
    Force unicode output

--value-output, -V
    Output JSON value (-Vr for raw string)

--version, -v
    Show version

CONFIGURATION

  Init files

To add your own functions you can use init.jq that is read from:

macOS
    $HOME/Library/Application Support/fq/init.jq or
    $HOME/.config/fq/init.jq

Linux and BSD
    $HOME/.config/fq/init.jq

Windows
    %AppData%\fq\init.jq

  Environment

NO_COLOR
    Non-empty string disables color output.

CLIUNICODE
    Enables use of unicode output characters.

COMPLETION_TIMEOUT
    REPL completion timeout in seconds.

NO_DECODE_PROGRESS
    Disables decode progress indicator.

EXIT STATUS

0 Success
2 Argument error or input IO error
3 Expression compile error
4 Decode error
5 Expression error

EXPRESSION

  Syntax

See jq(1) for syntax details. But here are some common beginner gotchas:

    - Functions that take no arguments are called using name instead of name().
    - Arguments are separated by ; instead of ,. Comma is used to concatenate output streams. To call a function f with two arguments use f(1; 2). If you do f(1, 2) you pass a single argument 1, 2, a filter that outputs 1 and then 2, to f.
    - Expressions can return or "output" zero or more values. This is how iteration etc is done, 1, 2 outputs 1 then 2.
    - Similar to shell pipelines, implicit input and output are used and piped together using |. . is used to refer to current input. Ex 1 | . + 2 outputs 3, 1, 2 | . + 2 outputs 3 and 4.
    - In the jq manual and other jq related documentation you might see name/2, this means the function name takes two arguments (arity).

  Additional features

fq uses an extended variant of the jq language with a few extra features:

    - Arbitrary-precision integers and arithmetic.
    - Supports raw strings using back-ticks. String interpolation and codepoint escapes are not processed.

        `hello \("world")\ud83c\udf0d`

    results in the string hello \("world")\ud83c\udf0d or as JSON "hello \\(\"world\")\\ud83c\\udf0d". In contrast

        "hello \("world")\ud83c\udf0d"

    results in the string hello world🌍 or as JSON "hello world🌍".
    - Supports more number bases in integer literals.
        - 0xabcd (Hexadecimal).
        - 0o125715 (Octal).
        - 0b1010101111001101 (Binary).
        - Grouping using underscore 0xab_cd.
    - Binary and decode value types, see below.
    - Try include using an ending question mark include "file?"; that doesn’t fail if file is missing or has errors.
    - Some values can act as an object with keys even when they are arrays, numbers etc.
    - There can be keys hidden from keys and [].
    - Some values are readonly and can’t be updated or will convert to JSON on update.
    - Mixing --args and --jsonargs does not behave the same as in jq.

  Additional functions

band, bor, bxor, bsl, bsr, bnot
    Bitwise operations as functions. Work the same as jq’s math functions. Functions that take one argument use input, 1 | bnot, and functions with more arguments ignore the input and use formal arguments bsl(1; 3).

chunk($size)
    Split array or string into $size length chunks. Last chunk might be shorter.

count, count_by(f)
    Like group but outputs array of [value, count] pairs.

delta, delta_by(f)
    Array with differences between consecutive elements. delta is the same as delta_by(.b - .a).

diff($a; $b)
    Produce a diff between $a and $b. Differences are represented as an object {a: <value from a>, b: <value from b>}.

expr_to_path
    Converts from a string ".key[1]" to a path value ["key", 1].

grep_by(f)
    Recursively select using a filter and ignore any errors. Ex: grep_by(. > 180 and . < 200), first(grep_by(format == "id3v2")). This is the same as doing .. | select(f)?.

group
    Group values, same as group_by(.).

path_to_expr
    Converts a path value ["key", 1] to a string ".key[1]".

paste
    Read string from stdin until ^D. Useful for pasting text. Ex: paste | from_pem | asn1_ber | repl read from stdin then decode and start a new sub-REPL with result.

streaks, streaks_by(f)
    Like group but groups streaks based on condition.

repl, repl($opts)
    Nested REPL. Must be last in a pipeline. repl can "slurp" outputs, ex: 1, 2, 3 | repl, and supports options, ex: [1,2,3] | repl({compact: true}).

slurp("<name>")
    Slurp outputs and saves them to $name. Must be last in the pipeline. Will be available as a global array $name. Ex 1,2,3 | slurp("a"), $a[] same as spew("a").

spew, spew("<name>")
    Outputs all or a specific slurp. Ex: spew("a").

println, print
    Print string or compact JSON to stdout with and without new line.

printerrln, printerr
    Print string or compact JSON to stderr with and without new line.

hexdump, hexdump($opts), hd, hd($opts)
    Display binary as a hex dump.

format
    Format name for a decode value. Ex: first(grep_by(format == "jpeg")).

  Decode value

A decode value is the type returned from decoding a format and used to represent values produced by a decoder. It can be seen as representing any standard jq type but with some additional properties attached.

Each decode value has these properties:

    - Bit range in the input. Can be used as a binary using tobytes, tobytesrange, tobits and tobitsrange.
    - If scalar type, an actual value:
        - This is the decoded representation of the bits, a number, string, bool etc.
        - Can be accessed using toactual.
    - If scalar type, an optional symbolic value:
        - Is usually a mapping of the actual to symbolic value, ex: map number to a string value.
        - Can be accessed using tosym.
    - An optional description:
        - Can be accessed using todescription.
    - parent is the parent decode value
    - parents is all the parent decode values
    - topath is the jq path for the decode value
    - torepr converts decode value to its representation if possible

The value of a decode value is the symbolic value if available and otherwise the actual value. To explicitly access the value use tovalue. In most expressions this is not needed as it will be done automatically.

  Decode value functions

root
    Root decode value for decode value.

buffer_root
    Root decode value of sub buffer for decode value.

format_root
    Root decode value of nested format for decode value.

parent
    Parent decode value for decode value.

parents
    Outputs all parent decode values from decode value.

topath
    Path for decode value. Use path_to_expr to get a string representation.

tovalue, tovalue($opts)
    Symbolic, if available, or actual value for decode value.

toactual, toactual($opts)
    Actual value for decode value.

tosym, tosym($opts)
    Symbolic value for decode value.

todescription
    Description for decode value.

torepr
    Converts decode value into what it represents. For example converts msgpack decode value into a value representing its JSON representation.

decode, decode($name), decode($name; $opts)
    Decode binary as a format. $name is format name, $opts is format options. Ex: ... | decode("mp4"). Note that all decoders are available as functions. Ex: ... | mp4 is the same as ... | decode("mp4").

tobytes, tobytesrange, tobits and tobitsrange on a decode value will return the raw source bits as a binary.

  Binary

Binary type is used to store raw bits or bytes. Raw bits will act as zero bits padded strings in standard jq expressions.

Use tobits and tobytes to create them from decode value, string, number or binary array. tobytes will if needed zero pad most significant bits to be byte aligned.

There is also tobitsrange and tobytesrange which do the same thing but will preserve source range when displayed.

    - "string" | tobytes produces a binary with UTF8 bytes.
    - 1234 | tobits produces a binary with the unsigned big-endian integer 1234 with enough bits to represent the number. Use tobytes to get the same but with enough bytes to represent the number. This is different to how numbers work inside binary arrays where they are limited to 0-255.
    - ["abc", 123, ...] | tobytes produces a binary from a binary array. See Binary array below.
    - .[index] accesses bit or byte at index index. Index is in units.
    - [0x12, 0x34, 0x56] | tobytes[1] is 0x34
    - [0x12, 0x34, 0x56] | tobits[3] is 1
    - .[start:], .[start:end] or .[:end] is normal jq slice syntax and will slice the binary from start to end. start and end are in units.
    - [0x12, 0x34, 0x56] | tobytes[1:2] will be a binary with the byte 0x34
    - [0x12, 0x34, 0x56] | tobits[4:12] will be a binary with the byte 0x23
    - [0x12, 0x34, 0x56] | tobits[4:20] will be a binary with the bytes 0x23, 0x45
    - [0x12, 0x34, 0x56] | tobits[4:20] | tobytes[1:] will be a binary with the byte 0x45
    - Both .[index] and .[start:end] support negative indices to index from end.
    - explode outputs an array with all bytes or bits as integers.

  Binary functions

grep($v), grep($v; $flags), vgrep($v), vgrep($v; $flags), bgrep($v), bgrep($v; $flags)
    Recursively match $v. $v is a scalar to match, where a string is treated as a regexp. A binary will match exact bytes. $flags arguments are regexp flags with additional flag "b" that will treat each byte in the input binary as a code point. This makes it possible to match exact bytes.

fgrep($v), fgrep($v; $flags)
    Recursively match field name in a decode value.

tobits
    Transform input to binary with bit as unit and don’t preserve source range.

tobitsrange
    Transform input to binary with bit as unit and preserve source range.

tobytes
    Transform input to binary with byte as unit and don’t preserve source range.

tobytesrange
    Transform input to binary with byte as unit and preserve source range.

open
    Open file for reading. Returns a binary. Ex: "file" | open | decode, "file" | open | tostring.

The standard jq functions test, match, capture, scan and split also work on binary values, matching against the raw bytes.

  Binary array

Binary array is a value "shape" and not a new type. It’s an array of numbers, strings, binaries or other binary arrays. They can be used as input to tobits, tobytes or other functions that accept a binary as input.

    - Number is a byte with value 0-255
    - String as UTF8 bytes
    - Binary as is
    - Binary array used recursively

Binary arrays are similar to and inspired by

Some examples:

    - [0, 123, 255] | tobytes will be binary with 3 bytes 0, 123 and 255.
    - [0, [123, 255]] | tobytes same as above.
    - [0, 1, 1, 0, 0, 1, 1, 0 | tobits] | tobytes will be binary with 1 byte, 0x66.
    - [(.a | tobytes[-10:]), 255, (.b | tobits[:10])] | tobytes the concatenation of the last 10 bytes of .a, byte of value 255 and the first 10 bits of .b.

  Differences to jq

See

  Naming inconsistencies

jq’s naming convention is a bit inconsistent. Some standard library functions are named tojson while others from_entries. fq follows this tradition but tries to use snake_case unless there is a good reason.

Here are all the non-snake_case functions added by fq. Most of them deal with decode and binary values which are new "primitive" types:

    - toactual
    - tobits
    - tobitsrange
    - tobytes
    - tobytesrange
    - todescription
    - topath
    - torepr
    - tosym
    - tovalue

DISPLAY OUTPUT

display or d is the main function for displaying values and is also the function that will be used if no other output function is explicitly used. If its input is a decode value it will output a dump and tree structure or otherwise it will output as JSON.

Below are some usage examples:

The first and second examples do the same thing, inputting "hello" to display.

    $ fq -n '"hello"'
    "hello"
    $ fq -n '"hello" | d'
    "hello"

In the next few examples we select out the first "edit list" box in an mp4 file and display it in various ways.

By default, display will only show the root level:

    $ fq 'first(grep_by(.type == "elst"))' file.mp4
         |00 01 02 03 04 05 06 07 08 09|0123456789|.boxes[3].boxes[1].boxes[1].boxes[0]{}: box
    0xd5c|                           00|         .|  size: 28
    0xd66|00 00 1c                     |...       |
    0xd66|         65 6c 73 74         |   elst   |  type: "elst" (An edit list)
    0xd66|                     00      |       .  |  version: 0
    0xd66|                        00 00|        ..|  flags: 0
    0xd70|00                           |.         |
    0xd70|   00 00 00 01               | ....     |  entry_count: 1
    0xd70|               00 00 00 28 00|     ...(.|  entries[0:1]:
    0xd7a|00 00 00 00 01 00 00         |.......   |

First row shows a ruler with byte offset into the line and jq path for the value.

The columns are:

    - Start address for the line. For example we see that size starts at 0xd5c (row) + 0x09 (column) = 0xd65.
    - Hex representation of input bits for value. Will show the whole byte even if the value only partially uses bits from it.
    - ASCII representation of input bits for value. Will show the whole byte even if the value only partially uses bits from it.
    - Tree structure of decoded value, symbolic value and description.

Notation:

    - {} value is an object that might have nested values.
    - [start:end] value is an array with index starting at start and ending at end (exclusive).

With display or d it will recursively show the whole tree:

    $ fq 'first(grep_by(.type == "elst")) | d' file.mp4
         |00 01 02 03 04 05 06 07 08 09|0123456789|.boxes[3].boxes[1].boxes[1].boxes[0]{}: box
    0xd5c|                           00|         .|  size: 28
    0xd66|00 00 1c                     |...       |
    0xd66|         65 6c 73 74         |   elst   |  type: "elst" (An edit list)
    0xd66|                     00      |       .  |  version: 0
    0xd66|                        00 00|        ..|  flags: 0
    0xd70|00                           |.         |
    0xd70|   00 00 00 01               | ....     |  entry_count: 1
         |                             |          |  entries[0:1]:
         |                             |          |    [0]{}: entry
    0xd70|               00 00 00 28   |     ...( |      segment_duration: 40
    0xd70|                           00|         .|      media_time: 0
    0xd7a|00 00 00                     |...       |
    0xd7a|         00 01 00 00         |   ....   |      media_rate: 1

Same but verbose dv:

    $ fq 'first(grep_by(.type == "elst")) | dv' file.mp4
         |00 01 02 03 04 05 06 07 08 09|0123456789|.boxes[3].boxes[1].boxes[1].boxes[0]{}: box 0xd65-0xd81 (28)
    0xd5c|                           00|         .|  size: 28 0xd65-0xd69 (4)
    0xd66|00 00 1c                     |...       |
    0xd66|         65 6c 73 74         |   elst   |  type: "elst" (An edit list) 0xd69-0xd6d (4)
    0xd66|                     00      |       .  |  version: 0 0xd6d-0xd6e (1)
    0xd66|                        00 00|        ..|  flags: 0 0xd6e-0xd71 (3)
    0xd70|00                           |.         |
    0xd70|   00 00 00 01               | ....     |  entry_count: 1 0xd71-0xd75 (4)
         |                             |          |  entries[0:1]: 0xd75-0xd81 (12)
         |                             |          |    [0]{}: entry 0xd75-0xd81 (12)
    0xd70|               00 00 00 28   |     ...( |      segment_duration: 40 0xd75-0xd79 (4)
    0xd70|                           00|         .|      media_time: 0 0xd79-0xd7d (4)
    0xd7a|00 00 00                     |...       |
    0xd7a|         00 01 00 00         |   ....   |      media_rate: 1 0xd7d-0xd81 (4)

In verbose mode bit ranges and array element names are shown.

Bit ranges use <start-byte>[.<bits>]-<end-byte>[.<bits>] as notation where .<bits> is left out if byte aligned. For example type starts at byte 0xd69 bit 0 (.0 is left out) and ends at 0xd6d bit 0 (exclusive) and has a size of 4 bytes.

This verbosely displays the header of the second frame in an mp3 file which has a bunch of non-byte-aligned fields:

    $ fq '.frames[1].header | dv' file.mp3
         |00 01 02 03 04 05 06 07 08 09|0123456789|.frames[1].header{}: 0xb79-0xb7d (4)
    0xb72|                     ff fb   |       .. |  sync: 0b11111111111 (valid) 0xb79-0xb7a.3 (1.3)
    0xb72|                        fb   |        . |  mpeg_version: "1" (3) (MPEG Version 1) 0xb7a.3-0xb7a.5 (0.2)
    0xb72|                        fb   |        . |  layer: 3 (1) (MPEG Layer 3) 0xb7a.5-0xb7a.7 (0.2)
         |                             |          |  sample_count: 1152
    0xb72|                        fb   |        . |  protection_absent: true (No CRC) 0xb7a.7-0xb7b (0.1)
    0xb72|                           50|         P|  bitrate: 64000 (5) 0xb7b-0xb7b.4 (0.4)
    0xb72|                           50|         P|  sample_rate: 44100 (0) 0xb7b.4-0xb7b.6 (0.2)
    0xb72|                           50|         P|  padding: "not_padded" (0b0) 0xb7b.6-0xb7b.7 (0.1)
    0xb72|                           50|         P|  private: 0 0xb7b.7-0xb7c (0.1)
    0xb7c|c4                           |.         |  channels: "mono" (0b11) 0xb7c-0xb7c.2 (0.2)
    0xb7c|c4                           |.         |  channel_mode: "none" (0b0) 0xb7c.2-0xb7c.4 (0.2)
    0xb7c|c4                           |.         |  copyright: 0 0xb7c.4-0xb7c.5 (0.1)
    0xb7c|c4                           |.         |  original: 1 0xb7c.5-0xb7c.6 (0.1)
    0xb7c|c4                           |.         |  emphasis: "none" (0b0) 0xb7c.6-0xb7d (0.2)

Here the sync pattern starts at 0xb79 (bit 0) and ends at 0xb7a.3 (exclusive) and has a size of 1 byte and 3 bits, 11 bits in total (8+3).

There are also some other display aliases:

    - da is display({array_truncate: 0, string_truncate: 0}) don’t truncate arrays and strings.
    - dd is display({array_truncate: 0, string_truncate: 0, display_bytes: 0}) don’t truncate arrays and strings, show all raw bytes.
    - dv is display({array_truncate: 0, string_truncate: 0, verbose: true}) don’t truncate arrays and strings and display verbosely.
    - ddv is display({array_truncate: 0, string_truncate: 0, display_bytes: 0, verbose: true}) don’t truncate arrays and strings, show all raw bytes and display verbosely.

FORMATS

By default fq will try to automatically determine input format. In some cases this might fail or is not possible, then a format can be specified using -d NAME. It’s possible sometimes to force decode and get a partial or broken result using -o force=true.

    # decode as msgpack
    $ fq -d msgpack d file
    # force decode as msgpack
    $ fq -d msgpack -o force=true d file
    # see msgpack format help
    $ fq -h msgpack
    # list supported formats
    $ fq -h formats

  Format options

Some formats have their own options that can be set using -o. For example the mp4 format has a decode_samples option that controls if individual samples should be decoded. To disable it one can do fq -o decode_samples=false . file.mp4. See format list for options.

  Format functions

In addition to using -d all format decoders are also available as normal jq functions. Each format provides multiple functions:

<name>
    Decode and return a decode value even on error. Ex: ... | mp4

<name>($options)
    Same as above with format options. Ex: ... | mp4({decode_samples: false})

from_<name>
    Decode or throw on error. Ex: ... | from_mp4

from_<name>($options)
    Same as above with format options. Ex: ... | from_mp4({decode_samples: false})

Example usage:

    # decode jpeg found inside some other format
    $ fq '.some[].query | jpeg' file
    # decode jpeg at byte range 100-200
    $ fq -d bytes '.[100:200] | jpeg' file

  Supported formats

aac_frame
    Advanced Audio Coding frame

    Options

    -o object_type=1
        Audio object type

adts
    Audio Data Transport Stream

adts_frame
    Audio Data Transport Stream frame

aiff
    Audio Interchange File Format

amf0
    Action Message Format 0

apev2
    APEv2 metadata tag

apple_bookmark
    Apple BookmarkData

    Apple’s bookmarkData format is used to encode information that can be resolved into a URL object for a file even if the user moves or renames it. Can also contain security scoping information for App Sandbox support. These bookmarkData blobs are often found encoded in data fields of Binary Property Lists. Notable examples include:
        - com.apple.finder.plist - contains an FXRecentFolders value, which is an array of ten objects, each of which consists of a name and file-bookmark field, which is a bookmarkData object for each recently accessed folder location.
        - com.apple.LSSharedFileList.RecentApplications.sfl2 - sfl2 files are actually plist files of the NSKeyedArchiver format. They can be parsed the same as plist files, but they have a more complicated tree-like structure than would typically be found, which can make locating and retrieving specific values difficult, even once it has been converted to a JSON representation. For more information about these types of files, see Sarah Edwards' excellent research on the subject (link in references).

    The grep_by function can be used to recursively descend through the decoded tree, probing for and selecting any bookmark blobs, then converting them to readable JSON with torepr:

        fq 'grep_by(.type=="data" and .value[0:4] == "book") | .value | apple_bookmark |
        torepr' <sfl2 file>

    Authors
            - David McDonald

    References
            -
            -
            -
            -

ar
    Unix archive

asn1_ber
    ASN1 BER (basic encoding rules, also CER and DER)

    Supports decoding BER, CER and DER (X.690).
        - Currently no extra validation is done for CER and DER.
        - Does not support specifying a schema.
        - Supports torepr but without schema all sequences and sets will be arrays.

    Can be used to decode certificates etc

        $ fq -d bytes 'from_pem | asn1_ber | d' cert.pem

    Can decode nested values

        $ fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber

    Manual schema

        $ fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der

    References
            -
            -
            -
            -

av1_ccr
    AV1 Codec Configuration Record

av1_frame
    AV1 frame

av1_obu
    AV1 Open Bitstream Unit

avc_annexb
    H.264/AVC Annex B

avc_au
    H.264/AVC Access Unit

    Options

    -o bottom_field_pic_order_in_frame_present_flag=false
        No description

    -o cpb_cnt=0
        No description

    -o cpb_removal_delay_length=0
        No description

    -o delta_pic_order_always_zero_flag=false
        No description

    -o dpb_output_delay_length=0
        No description

    -o frame_mbs_only_flag=true
        No description

    -o initial_cpb_removal_delay_length=0
        No description

    -o length_size=0
        Length value size

    -o log2max_frame_num=4
        No description

    -o log2max_pic_order_cnt_lsb=4
        No description

    -o nal_hrd_parameters_present=false
        No description

    -o pic_order_cnt_type=0
        No description

    -o redundant_pic_cnt_present_flag=false
        No description

    -o separate_colour_plane_flag=false
        No description

    -o time_offset_length=0
        No description

    -o vcl_hrd_parameters_present=false
        No description

avc_dcr
    H.264/AVC Decoder Configuration Record

avc_nalu
    H.264/AVC Network Access Layer Unit

    Options

    -o bottom_field_pic_order_in_frame_present_flag=false
        No description

    -o cpb_cnt=0
        No description

    -o cpb_removal_delay_length=0
        No description

    -o delta_pic_order_always_zero_flag=false
        No description

    -o dpb_output_delay_length=0
        No description

    -o frame_mbs_only_flag=true
        No description

    -o initial_cpb_removal_delay_length=0
        No description

    -o log2max_frame_num=4
        No description

    -o log2max_pic_order_cnt_lsb=4
        No description

    -o nal_hrd_parameters_present=false
        No description

    -o pic_order_cnt_type=0
        No description

    -o redundant_pic_cnt_present_flag=false
        No description

    -o separate_colour_plane_flag=false
        No description

    -o time_offset_length=0
        No description

    -o vcl_hrd_parameters_present=false
        No description

avc_pps
    H.264/AVC Picture Parameter Set

avc_sei
    H.264/AVC Supplemental Enhancement Information

    Options

    -o cpb_cnt=0
        No description

    -o cpb_removal_delay_length=0
        No description

    -o delta_pic_order_always_zero_flag=false
        No description

    -o dpb_output_delay_length=0
        No description

    -o frame_mbs_only_flag=true
        No description

    -o initial_cpb_removal_delay_length=0
        No description

    -o log2max_frame_num=4
        No description

    -o log2max_pic_order_cnt_lsb=4
        No description

    -o nal_hrd_parameters_present=false
        No description

    -o pic_order_cnt_type=0
        No description

    -o separate_colour_plane_flag=false
        No description

    -o time_offset_length=0
        No description

    -o vcl_hrd_parameters_present=false
        No description

avc_sps
    H.264/AVC Sequence Parameter Set

avi
    Audio Video Interleaved

    Samples
        AVI has many redundant ways to index samples so currently .streams[].samples will only include samples using the most "modern" method found in the file. That is in order of stream super index, movi ix index then idx1 index.

    Extract samples for stream 1

        $ fq '.streams[1].samples[] | tobytes' file.avi > stream01.mp3

    Show stream summary

        $ fq -o decode_samples=false '[.chunks[0] | grep_by(.id=="LIST" and .type=="strl") | grep_by(.id=="strh") as {$type} | grep_by(.id=="strf") as {$format_tag, $compression} | {$type,$format_tag,$compression}]' *.avi

    Speed up decoding by disabling sample and extended chunks decoding
        If you’re not interested in sample details or extended chunks you can speed up decoding by using:

        $ fq -o decode_samples=false -o decode_extended_chunks=false d file.avi

    References
            -
            -

    Options

    -o decode_extended_chunks=true
        Decode extended chunks

    -o decode_samples=true
        Decode samples

avro_ocf
    Avro object container file

    Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification. Capable of handling null, deflate, and snappy codecs for data compression. Limitations:
        - Schema does not support self-referential types, only built-in types.
        - Decimal logical types are not supported for decoding, will just be treated as their primitive type

    References
            -

    Authors
            - Xentripetal

bencode
    BitTorrent bencoding

    Convert represented value to JSON

        $ fq -d bencode torepr file.torrent

    References
            -

bitcoin_blkdat
    Bitcoin blk.dat

bitcoin_block
    Bitcoin block

    Options

    -o has_header=false
        Has blkdat header

bitcoin_script
    Bitcoin script

bitcoin_transaction
    Bitcoin transaction

bits
    Raw bits

    Decode to a slice and indexable binary of bits.

    Slice and decode bit range

        $ echo 'some {"a":1} json' | fq -d bits '.[40:-48] | fromjson'
        {
          "a": 1
        }

    Index bits

        $ echo 'hello' | fq -d bits '.[4]'
        1
        $ echo 'hello' | fq -c -d bits '[.[range(8)]]'
        [0,1,1,0,1,0,0,0]

bplist
    Apple Binary Property List

    Show full decoding

        $ fq d Info.plist

    Timestamps
        Timestamps in Apple Binary Property Lists are encoded as Cocoa Core Data timestamps, where the raw value is the floating point number of seconds since January 1, 2001. By default, fq will render the raw floating point value. In order to get the raw value or the string description, use the tovalue or todescription functions:

        $ fq 'torepr.SomeTimeStamp | tovalue' Info.plist
        685135328
        $ fq 'torepr.SomeTimeStamp | todescription' Info.plist
        "2022-09-17T19:22:08Z"

    Get JSON representation
        bplist files can be converted to a JSON representation using the torepr filter:

        $ fq torepr com.apple.UIAutomation.plist
        {
          "UIAutomationEnabled": true
        }

    Decoding NSKeyedArchiver serialized objects
        A common way that Swift and Objective-C libraries on macOS serialize objects is through the NSKeyedArchiver API, which flattens objects into a list of elements and class descriptions that are reconstructed into an object graph using CFUID elements in the property list. fq includes a function, from_ns_keyed_archiver, which will rebuild this object graph into a friendly representation. If no parameters are supplied, it will assume that there is a CFUID located at ."$top".root that specifies the root from which decoding should occur. If this is not present, an error will be produced, asking the user to specify a root object in the .$objects list from which to decode. The following examples show how this might be used (in this case, within the fq REPL):

        # Assume $top.root is present
        bplist> from_ns_keyed_archiver
        # Specify optional root
        bplist> from_ns_keyed_archiver(1)

    Authors
            - David McDonald

    References
            -
            -
            -

bsd_loopback_frame
    BSD loopback frame

bson
    Binary JSON

    Limitations
            - The decimal128 type is not supported for decoding, will just be treated as binary

    Convert represented value to JSON

        $ fq -d bson torepr file.bson

    Filter represented value

        $ fq -d bson 'torepr | select(.name=="bob")' file.bson

    Authors
            - Mattias Wadman original author
            - Matt Dale additional types and bug fixes

    References
            -

bytes
    Raw bytes

    Decode to a slice and indexable binary of bytes.

    Slice out byte ranges

        $ echo -n 'hello' | fq -d bytes '.[-3:]' > last_3_bytes
        $ echo -n 'hello' | fq -d bytes '[.[-2:], .[0:2]] | tobytes' > first_last_2_bytes_swapped

    Slice and decode byte range

        $ echo 'some {"a":1} json' | fq -d bytes '.[5:-6] | fromjson'
        {
          "a": 1
        }

    Index bytes

        $ echo 'hello' | fq -d bytes '.[1]'
        101

bzip2
    bzip2 compression

caff
    Live2D Cubism archive

    Authors
            -

    Options

    -o uncompress=true
        Uncompress and probe files

cbor
    Concise Binary Object Representation

    Convert represented value to JSON

        $ fq -d cbor torepr file.cbor

    References
            -
            -

csv
    Comma separated values

    TSV to CSV

        $ fq -d csv -o comma="\t" to_csv file.tsv

    Convert rows to objects based on header row

        $ fq -d csv '.[0] as $t | .[1:] | map(with_entries(.key = $t[.key]))' file.csv

    Options

    -o comma=","
        Separator character

    -o comment="#"
        Comment line character

dns
    DNS packet

dns_tcp
    DNS packet (TCP)

elf
    Executable and Linkable Format

ether8023_frame
    Ethernet 802.3 frame

exif
    Exchangeable Image File Format

fairplay_spc
    FairPlay Server Playback Context

fit
    Garmin Flexible and Interoperable Data Transfer

    Limitations
            - Fields with subcomponents, such as "compressed_speed_distance" field on globalMessageNumber 20 is not represented correctly. The field is read as 3 separate bytes where the first 12 bits are speed and the last 12 bits are distance.
            - There are still lots of UNKNOWN fields due to gaps in Garmin’s SDK Profile documentation. (Currently FIT SDK 21.126)
            - Compressed timestamp messages are not accumulated against last known full timestamp.

    Convert stream of data messages to JSON array

        $ fq '[.data_records[] | select(.record_header.message_type == "data").data_message]' file.fit

    Authors
            - Mikael Lofjärd original author

    References
            -
            -

flac
    Free Lossless Audio Codec file

flac_frame
    FLAC frame

    Options

    -o bits_per_sample=16
        Bits per sample

    -o sample_details=false
        Decode more sample details like residuals etc

flac_metadatablock
    FLAC metadatablock

flac_metadatablocks
    FLAC metadatablocks

flac_picture
    FLAC metadatablock picture

flac_streaminfo
    FLAC streaminfo

gif
    Graphics Interchange Format

gzip
    gzip compression

heif
    High Efficiency Image Format

    Options

    -o allow_truncated=false
        Allow box to be truncated

hevc_annexb
    H.265/HEVC Annex B

hevc_au
    H.265/HEVC Access Unit

    Options

    -o length_size=4
        Length value size

hevc_dcr
    H.265/HEVC Decoder Configuration Record

hevc_nalu
    H.265/HEVC Network Access Layer Unit

hevc_pps
    H.265/HEVC Picture Parameter Set

hevc_sps
    H.265/HEVC Sequence Parameter Set

hevc_vps
    H.265/HEVC Video Parameter Set

html
    HyperText Markup Language

    HTML is decoded in HTML5 mode and will always include <html>, <body> and <head> elements. See xml format for more examples and how to preserve element order and how to encode to xml. There is no to_html function, see to_xml instead.

    Element as object

        # decode as object is the default
        $ echo '<a href="url">text</a>' | fq -d html
        {
          "html": {
            "body": {
              "a": {
                "#text": "text",
                "@href": "url"
              }
            },
            "head": ""
          }
        }

    Element as array

        $ echo '<a href="url">text</a>' | fq -d html -o array=true
        [
          "html",
          null,
          [
            [
              "head",
              null,
              []
            ],
            [
              "body",
              null,
              [
                [
                  "a",
                  {
                    "#text": "text",
                    "href": "url"
                  },
                  []
                ]
              ]
            ]
          ]
        ]
        # decode html files to a {file: "title", ...} object
        $ fq -n -d html '[inputs | {key: input_filename, value: .html.head.title?}] | from_entries' *.html
        # <a> href:s in file
        $ fq -r -o array=true -d html '.. | select(.[0] == "a" and .[1].href)?.[1].href' file.html

    Options

    -o array=false
        Decode as nested arrays

    -o attribute_prefix="@"
        Prefix for attribute keys

    -o seq=false
        Use seq attribute to preserve element order

icc_profile
    International Color Consortium profile

icmp
    Internet Control Message Protocol

icmpv6
    Internet Control Message Protocol v6

id3v1
    ID3v1 metadata

id3v11
    ID3v1.1 metadata

id3v2
    ID3v2 metadata

ipv4_packet
    Internet protocol v4 packet

ipv6_packet
    Internet protocol v6 packet

jp2c
    JPEG 2000 codestream

jpeg
    Joint Photographic Experts Group file

json
    JavaScript Object Notation

jsonl
    JavaScript Object Notation Lines

leveldb_descriptor
    LevelDB Descriptor

    Limitations
            - fragmented non-"full" records are not merged and decoded further.

    Authors
            - original author

    References
            -
            -
            -

leveldb_log
    LevelDB Log

    Limitations
            - fragmented non-"full" records are not merged and decoded further.

    Authors
            - original author

    References
            -
            -
            -

leveldb_table
    LevelDB Table

    Limitations
            - no Meta Blocks (like "filter") are decoded yet.
            - Zstandard decompression is not implemented yet.

    Authors
            - original author

    References
            -
            -
            -

luajit
    LuaJIT 2.0 bytecode

    Authors
            -

    References
            -
            -

macho
    Mach-O macOS executable

    Supports decoding vanilla and FAT Mach-O binaries.

    Select 64bit load segments

        $ fq '.load_commands[] | select(.cmd=="segment_64")' file

    References
            -

    Authors
            - Sıddık AÇIL

macho_fat
    Fat Mach-O macOS executable (multi-architecture)

markdown
    Markdown

    Array with all level 1 and 2 headers

        $ fq -d markdown '[.. | select(.type=="heading" and .level<=2)?.children[0]]' file.md

matroska
    Matroska file

    Lookup element using path

        $ fq 'matroska_path(".Segment.Tracks[0]")' file.mkv

    Get path to element

        $ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv

    References
            -
            -
            -
            -
            -

    Options

    -o decode_samples=true
        Decode samples

midi
    Standard MIDI file

    Notes
            - Only supports the MIDI 1.0 MIDI file specification.
            - Only supports MThd and MTrk chunks.
            - Does only basic validation on the MIDI data.

    Sample queries
            - Extract the track names from a MIDI file

        fq -d midi '.. | select(.event=="track_name")? | "\(.track_name)"' midi/twinkle.mid
        - Extract the tempo changes from a MIDI file

        fq -d midi '.. | select(.event=="tempo")?.tempo' midi/twinkle.mid
        - Extract the key changes from a MIDI file

        fq -d midi '.. | select(.event=="key_signature")?.key_signature' midi/twinkle.mid
        - Extract NoteOn events:

        fq -d midi 'grep_by(.event=="note_on") | [.time.tick, .note_on.note] | join(" ")' midi/twinkle.mid

    Authors
            -

    References
            -
            -
            -
            -
            -
            -

moc3
    MOC3 file

    Authors
            -

mp3
    MP3 file

    Options

    -o max_sync_seek=32768
        Max byte distance to next sync

    -o max_unique_header_configs=5
        Max number of unique frame header configs allowed

    -o max_unknown=50
        Max percent (0-100) unknown bits

mp3_frame
    MPEG audio layer 3 frame

mp3_frame_vbri
    MP3 frame Fraunhofer encoder variable bitrate tag

mp3_frame_xing
    MP3 frame Xing/Info tag

mp4
    ISOBMFF, QuickTime and similar

    Speed up decoding by not decoding samples

        # manually decode first sample as an aac_frame
        $ fq -o decode_samples=false '.tracks[0].samples[0] | aac_frame | d' file.mp4

    Entries for first edit list as values

        $ fq 'first(grep_by(.type=="elst").entries) | tovalue' file.mp4

    Whole box tree as JSON (exclude mdat data and tracks)

        $ fq 'del(.tracks) | grep_by(.type=="mdat").data = "<excluded>" | tovalue' file.mp4

    Force decode a single box

        $ fq -n '"AAAAHGVsc3QAAAAAAAAAAQAAADIAAAQAAAEAAA==" | from_base64 | mp4({force:true}) | d'

    Lookup mp4 box using an mp4 box path

        # <decode value box> | mp4_path($path) -> <decode value box>
        $ fq 'mp4_path(".moov.trak[1]")' file.mp4

    Get mp4 box path for a decode value box

        # <decode value box> | mp4_path -> string
        $ fq 'grep_by(.type == "trak") | mp4_path' file.mp4

    References
            -
            -

    Options

    -o allow_truncated=false
        Allow box to be truncated

    -o decode_samples=true
        Decode track samples

    -o skip_samples=false
        Skip track samples

mpeg_asc
    MPEG-4 Audio Specific Config

mpeg_es
    MPEG Elementary Stream

mpeg_pes
    MPEG Packetized elementary stream

mpeg_pes_packet
    MPEG Packetized elementary stream packet

mpeg_spu
    Sub Picture Unit (DVD subtitle)

mpeg_ts
    MPEG Transport Stream

msgpack
    MessagePack

    Convert represented value to JSON

        $ fq -d msgpack torepr file.msgpack

    References
            -

negentropy
    Negentropy message

    View a full Negentropy message

        $ fq -d negentropy dd file

    Or from hex

        $ echo *(Aq | fq -R 'from_hex | negentropy | dd'

    Check how many ranges the message has and how many of those are of 'fingerprint' mode

        $ fq -d negentropy '.bounds | length as $total | map(select(.mode == "fingerprint")) | length | {$total, fingerprint: .}' message

    Get all ids in all idlists

        $ fq -d negentropy '.bounds | map(select(.mode == "idlist") | .idlist | .ids) | flatten' message

    Authors
            - fiatjaf,

    References
            -

nes
    iNES/NES 2.0 cartridge ROM format

    Limitations
            - prg_rom, chr_rom and trainer fields may contain data that is just random junk from the memory chips, since they are of a fixed size.
            - The nes_toasm function outputs ALL opcodes, including the unofficial ones, which means that none of the regular assemblers can recompile it.
            - The nes_tokitty function works on tiles in chr_rom but only outputs a Kitty graphics compatible string. You need to manually printf that string to get Kitty (or another compatible terminal) to output the graphics.

    Decompile PRG ROM

        $ fq -r '.prg_rom[] | nes_toasm' file.nes

    Print out first CHR ROM tile in Kitty (or Konsole, wayst, WezTerm) at size 5

        $ printf $(fq -r -d nes '.chr_rom[0] | nes_tokitty(5)' file.nes)

    Print out all CHR ROM tiles in Kitty (with Bash) at size 5

        $ for line in $(fq -r '.chr_rom[] | nes_tokitty(5)' file.nes);do printf "%b%s" "$line";done

    Authors
            - Mikael Lofjärd original author

    References
            -
            -
            -
            -

ogg
    OGG file

ogg_page
    OGG page

opentimestamps
    OpenTimestamps file

    View a full OpenTimestamps file

        $ fq dd file.ots

    List the names of the Calendar servers used

        $ fq '.operations | map(select(.attestation_type == "calendar") | .url)' file.ots

    Check if there are Bitcoin attestations present

        $ fq '.operations | map(select(.attestation_type == "bitcoin")) | length > 0' file.ots

    Authors
            - fiatjaf,

    References
            -
            -

opus_packet
    Opus packet

pcap
    PCAP packet capture

    Build object with number of (reassembled) TCP bytes sent to/from client IP

        # for a pcapng file you would use .[0].tcp_connections for first section
        $ fq '.tcp_connections | group_by(.client.ip) | map({key: .[0].client.ip, value: map(.client.stream, .server.stream | tobytes.size) | add}) | from_entries'
        {
          "10.1.0.22": 15116,
          "10.99.12.136": 234,
          "10.99.12.150": 218
        }

pcapng
    PCAPNG packet capture

pg_btree
    PostgreSQL btree index file

    Btree index meta page

        $ fq -d pg_btree -o flavour=postgres14 ".[0] | d" 16404

    Btree index page

        $ fq -d pg_btree -o flavour=postgres14 ".[1]" 16404

    Authors
            - Pavel Safonov

    References
            -

    Options

    -o page=0
        First page number in file, default is 0

pg_control
    PostgreSQL control file

    Decode content of pg_control file

        $ fq -d pg_control -o flavour=postgres14 d pg_control

    Specific fields can be queried individually

        $ fq -d pg_control -o flavour=postgres14 ".state, .check_point_copy.redo, .wal_level" pg_control

    Authors
            - Pavel Safonov

    References
            -

    Options

    -o flavour=""
        PostgreSQL flavour: postgres14, pgproee14, postgres10

pg_heap
    PostgreSQL heap file

    To see heap page’s content

        $ fq -d pg_heap -o flavour=postgres14 ".[0]" 16994

    To see page’s header

        $ fq -d pg_heap -o flavour=postgres14 ".[0].page_header" 16994

    First and last item pointers on first page

        $ fq -d pg_heap -o flavour=postgres14 ".[0].pd_linp[0, -1]" 16994

    First and last tuple on first page

        $ fq -d pg_heap -o flavour=postgres14 ".[0].tuples[0, -1]" 16994

    Authors
            - Pavel Safonov

    References
            -

    Options

    -o flavour="postgres14"
        PostgreSQL flavour: postgres14, pgproee14, postgres10

    -o page=0
        First page number in file, default is 0

    -o segment=0
        Segment file number (16790.1 is 1), default is 0

png
    Portable Network Graphics file

prores_frame
    Apple ProRes frame

protobuf
    Protobuf

    Can decode sub messages

        $ fq -d protobuf '.fields[6].wire_value | protobuf | d' file

    References
            -

protobuf_widevine
    Widevine protobuf

pssh_playready
    PlayReady PSSH

rtmp
    Real-Time Messaging Protocol

    Currently only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).

    Show rtmp streams in PCAP file

        fq '.tcp_connections[] | select(.server.port=="rtmp") | d' file.cap

    References
            -
            -

safetensors
    SafeTensors

sll2_packet
    Linux cooked capture encapsulation v2

sll_packet
    Linux cooked capture encapsulation

stl
    Standard Tessellation Language

    Decode binary STL (Stereolithography, Standard Tessellation Language) files.

    Current limitations
            - No support for ASCII STL files
            - No support for VisCAM and SolidView colors
            - No support for Materialise Magics colors

tap
    TAP tape format for ZX Spectrum computers

    The TAP- (and BLK-) format is nearly a direct copy of the data that is stored in real tapes, as it is written by the ROM save routine of the ZX-Spectrum. A TAP file is simply one data block or a group of 2 or more data blocks, one followed after the other. The TAP file may be empty. You will often find this format embedded inside the TZX tape format. The default file extension is .tap.

    Processing JSON files
        When needing to process a generated JSON file it’s recommended to convert the plain data bytes to an array by setting bits_format=byte_array:

        fq -o bits_format=byte_array -d tap -V d /path/to/file.tap

    Authors
            - Michael R. Cook original author

    References
            -

tar
    Tar archive

tcp_segment
    Transmission control protocol segment

tiff
    Tag Image File Format

tls
    Transport layer security

    Supports decoding of most standard records, messages and extensions. Can also decrypt most standard cipher suites in a PCAP with traffic in both directions if an NSS key log is provided.

    Decode and decrypt providing a PCAP and key log
        Write traffic to a PCAP file:

        $ tcpdump -i <iface> -w traffic.pcap

    Make sure your curl TLS backend supports SSLKEYLOGFILE and do:

        $ SSLKEYLOGFILE=traffic.keylog curl --tls-max 1.2 https://host/path

    Decode, decrypt and query. Uses keylog=@<path> to read option value from keylog file:

        # decode and show whole tree
        $ fq -o keylog=@traffic.keylog d traffic.pcap
        # write unencrypted server response to a file.
        # first .stream is the TCP stream, second .stream is TLS application data stream
        #
        # first TCP connections:
        $ fq -o keylog=@traffic.keylog '.tcp_connections[0].server.stream.stream | tobytes' traffic.pcap > data
        # first TLS connection:
        $ fq -o keylog=@traffic.keylog 'first(grep_by(.server.stream | format == "tls")).server.stream.stream | tobytes' > data

    Supported cipher suites for decryption
        TLS_DH_ANON_EXPORT_WITH_DES40_CBC_SHA, TLS_DH_ANON_EXPORT_WITH_RC4_40_MD5, TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_DES_CBC_SHA, TLS_DHE_DSS_WITH_RC4_128_SHA, TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_RSA_WITH_DES_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_RC4_128_SHA, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA, TLS_ECDHE_PSK_WITH_AES_128_GCM_SHA256, TLS_ECDHE_PSK_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_PSK_WITH_AES_128_CBC_SHA, TLS_PSK_WITH_AES_256_CBC_SHA, TLS_PSK_WITH_RC4_128_SHA, TLS_RSA_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_EXPORT_WITH_RC4_40_MD5, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_DES_CBC_SHA, TLS_RSA_WITH_RC4_128_MD5, TLS_RSA_WITH_RC4_128_SHA

    References
            -
            -

    Options

    -o keylog=""
        NSS Key Log content

toml
    Tom’s Obvious, Minimal Language

tzif
    Time Zone Information Format

    Get last transition time

        fq '.v2plusdatablock.transition_times[-1] | tovalue' tziffile

    Count leap second records

        fq '.v2plusdatablock.leap_second_records | length' tziffile

    Authors
            - Takashi Oguma

    References
            -

tzx
    TZX tape format for ZX Spectrum computers

    TZX is a file format designed to preserve cassette tapes compatible with the ZX Spectrum computers, although some specialized versions of the format have been defined for other machines such as the Amstrad CPC and C64. The format was originally created by Tomaz Kac, who was the maintainer until revision 1.13, before passing it to Martijn v.d. Heide. For a brief period the company Ramsoft became the maintainers, and created revision v1.20. The default file extension is .tzx.

    Processing JSON files
        When needing to process a generated JSON file it’s recommended to convert the plain data bytes to an array by setting bits_format=byte_array:

        fq -o bits_format=byte_array -d tzx -V d /path/to/file.tzx

    Authors
            - Michael R. Cook original author

    References
            -

udp_datagram
    User datagram protocol

vorbis_comment
    Vorbis comment

vorbis_packet
    Vorbis packet

vp8_frame
    VP8 frame

vp9_cfm
    VP9 Codec Feature Metadata

vp9_frame
    VP9 frame

vpx_ccr
    VPX Codec Configuration Record

wasm
    WebAssembly Binary Format

    Count opcode usage

        $ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' file.wasm

    List exports and imports

        $ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' file.wasm

    Authors
            - Takashi Oguma

    References
            -

wav
    WAV file

webp
    WebP image

xml
    Extensible Markup Language

    XML can be decoded and encoded into jq values in two ways, elements as object or array. The object variant might be easier to query for a specific value but array might be easier to use to generate xml or to query after all elements of some kind etc. Encoding is done using the to_xml function and it will figure out what variant is used based on the input value. It has two optional options indent and attribute_prefix.

    Elements as object
        Element can have different shapes depending on body text, attributes and children:
            - <a key="value">text</a> is {"a":{"#text":"text","@key":"value"}}, has text (#text) and attributes (@key)
            - <a>text</a> is {"a":"text"}
            - <a><b>text</b></a> is {"a":{"b":"text"}} one child with only text and no attributes
            - <a><b/><b>text</b></a> is {"a":{"b":["","text"]}} two children with same name end up in an array
            - <a><b/><b key="value">text</b></a> is {"a":{"b":["",{"#text":"text","@key":"value"}]}}

    If there is #seq attribute it encodes the child element order. Use -o seq=true to include sequence number when decoding, otherwise order might be lost.

        # decode as object is the default
        $ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -d xml -o seq=true
        {
          "a": {
            "b": [
              {
                "#seq": 0
              },
              {
                "#seq": 1,
                "#text": "bbb"
              }
            ],
            "c": {
              "#seq": 2,
              "#text": "ccc",
              "@attr": "value"
            }
          }
        }
        # access text of the <c> element
        $ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq '.a.c["#text"]'
        "ccc"
        # decode to object and encode to xml
        $ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -r -d xml -o seq=true 'to_xml({indent:2})'
        <a>
          <b></b>
          <b>bbb</b>
          <c attr="value">ccc</c>
        </a>

    Elements as array
        Elements are arrays of the shape ["name", null|{#text|attribute: "value"}, [<child element>, ...]].

        # decode as array
        $ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -d xml -o array=true
        [
          "a",
          null,
          [
            [
              "b",
              null,
              []
            ],
            [
              "b",
              {
                "#text": "bbb"
              },
              []
            ],
            [
              "c",
              {
                "#text": "ccc",
                "attr": "value"
              },
              []
            ]
          ]
        ]
        # decode to array and encode to xml
        $ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -r -d xml -o array=true -o seq=true 'to_xml({indent:2})'
        <a>
          <b></b>
          <b>bbb</b>
          <c attr="value">ccc</c>
        </a>
        # access text of the <c> element, the object variant above is probably easier to use
        $ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -o array=true '.[2][2][1]["#text"]'
        "ccc"

    References
            -

    Options

    -o array=false
        Decode as nested arrays

    -o attribute_prefix="@"
        Prefix for attribute keys

    -o seq=false
        Use seq attribute to preserve element order

yaml
    YAML Ain’t Markup Language

zip
    ZIP archive

    Supports ZIP64.

    Timestamp and time zones
        The timestamp accessed via .local_files[].last_modification is encoded in ZIP files using which lacks a known time zone. Probably the local time/date was used at creation. The unix_guess field in last_modification is a guess assuming the local time zone was UTC at creation.

    References
            -
            -
            -
            -

    Options

    -o uncompress=true
        Uncompress and probe files

ENCODINGS, SERIALIZATIONS AND HASHES

In addition to binary formats fq also supports various encodings and serialization formats.

At the moment fq does not have any dedicated argument for serialization formats but raw string input -R slurp -s and raw string output -r can make things easier. The combination -Rs will read all inputs into one string (same as jq).

Note that from* functions output jq values and to* functions take jq values as input so in some cases not all information will be properly preserved. For example, the element and attribute order might change and text and comment nodes might move or be merged. might be a better tool if that is needed.

Some example usages:

    # read yml (format is probed, use -d yaml to force) and do some query
    $ fq '...' file.yml
    # convert YAML to JSON
    # note the -r for raw string output, without it a JSON string with escaped JSON would be output
    $ fq -r 'tojson({indent:2})' file.yml
    # add token to URL
    $ echo -n "https://host.org" | fq -Rsr 'from_url | .user.username="token" | to_url'
    https://token@host.org
    # top 3 hosts in src or href attributes:
    # -d to decode as html, can't be probed as html5 parsers always produce some parse tree
    # [...] to start collecting values into an array
    # .. | ."@src"?, ."@href"? | values, recurse and try (?) to get src and href attributes and filter out nulls
    # from_url.host | values, parse as url and filter out those without a host
    # count to count unique values, returns [[key, count], ...]
    # reverse sort by count and pick first 3
    # map [key, count] tuples into {key: key, value: count}
    # from_entries, convert into object
    $ curl -s https://www.discogs.com/ | fq -d html '[.. | ."@src"?, ."@href"? | values | from_url.host | values] | count | sort_by(-.[1])[0:3] | map({key: .[0], value: .[1]}) | from_entries'
    {
      "blog.discogs.com": 9,
      "st.discogs.com": 10,
      "www.discogs.com": 14
    }
    # shows how serialization functions can be used on any string, how to transform values and output some other format
    # read and decode zip file and start an interactive REPL
    $ fq -i . <(curl -sL https://github.com/stefangabos/world_countries/archive/master.zip)
    # select from interesting xml file
    zip> .local_files[] | select(.file_name == "world_countries-master/data/countries/en/world.xml").uncompressed | repl
    # convert xml into jq value
    > .local_files[95].uncompressed string> from_xml | repl
    # sort countries by name and select the first one
    >> object> .countries.country | sort_by(."@name") | first | repl
    # see what current input is
    >>> object> .
    {
      "@alpha2": "af",
      "@alpha3": "afg",
      "@id": "4",
      "@name": "Afghanistan"
    }
    # remove "@" prefix from keys and convert to YAML and print it
    >>> object> with_entries(.key |= .[1:]) | to_yaml | print
    alpha2: af
    alpha3: afg
    id: "4"
    name: Afghanistan
    # exit all REPLs back to shell
    >>> object> ^D
    >> object> ^D
    > .local_files[95].uncompressed string> ^D
    zip> ^D

  XML and HTML

    - from_xml/from_xml($opts) Parse XML into jq value. $opts are:
        - {seq: true} preserve element ordering if more than one sibling.
        - {array: true} use nested [name, attributes, children] arrays to represent elements. Attributes will be null if none and children will be [] if none, this is to make it easier to work with as the array always has 3 values. to_xml does not require this.
    - from_html/from_html($opts) Parse HTML into jq value.
    Similar to from_xml but parses html5 in non-script mode. Will always have a html root with head and body elements. + $opts are:
        - {array: true} use nested arrays to represent elements.
        - {seq: true} preserve element ordering if more than one sibling.
    - to_xml/to_xml($opts) Serialize jq value into XML.
    Assumes object representation if input is an object, and nested arrays if input is an array.
    Will automatically add a root doc element if jq value has more than one root element.
    If a #seq is found on at least one element all siblings will be sorted by sequence number. Attributes are always sorted. + $opts are:
        - {indent: number} indent child elements.

XML elements can be represented as jq value in two ways, as objects (inspired by and or nested arrays. Both representations are lossy and might lose ordering of elements, text nodes and comments. In object representation from_xml, from_html and to_xml support {seq: true} option to parse/serialize {"#seq": <number>} attributes to preserve element sibling ordering.

The object version is denser and convenient to query, the nested arrays version is probably easier to use when generating XML.

Let’s assume $xml is this XML document as a string:

    <doc>
      <child attr="1"></child>
      <child attr="2">text</child>
      <other>text</other>
    </doc>

With object representation an element is represented as:

    - Attributes as @ prefixed @<key> keys.
    - Text nodes as #text.
    - Comment nodes as #comment keys.
    - For explicit sibling ordering #seq keys with a number, can be negative, assumed zero if missing.
    - Child element with only text as <name> key with text as value.
    - Child element with more than just text as <name> key with value an object.
    - Multiple child element siblings with same name as <name> key with value as array with strings and objects.

    > $xml | from_xml
    {
      "doc": {
        "child": [
          {
            "@attr": "1"
          },
          {
            "#text": "text",
            "@attr": "2"
          }
        ],
        "other": "text"
      }
    }

With nested array representation, an array with these values ["<name>", {attributes...}, [children...]].

    - Index 0 is an element name.
    - Index 1 object attributes (including #text and #comment keys).
    - Index 2 array of child elements.

    > $xml | from_xml({array: true})
    [
      "doc",
      null,
      [
        [
          "child",
          {
            "attr": "1"
          },
          []
        ],
        [
          "child",
          {
            "#text": "text",
            "attr": "2"
          },
          []
        ],
        [
          "other",
          {
            "#text": "text"
          },
          []
        ]
      ]
    ]

Parse and include #seq attributes if needed:

    > $xml | from_xml({seq:true})
    {
      "doc": {
        "child": [
          {
            "#seq": 0,
            "@attr": "1"
          },
          {
            "#seq": 1,
            "#text": "text",
            "@attr": "2"
          }
        ],
        "other": {
          "#seq": 2,
          "#text": "text"
        }
      }
    }

Select values in <doc>, remove <child>, add a <new> element, serialize to xml with 2 space indent and print the string

    > $xml | from_xml.doc | del(.child) | .new = "abc" | {root: .} | to_xml({indent: 2}) | println
    <root>
      <new>abc</new>
      <other>text</other>
    </root>

  JSON

    - fromjson Parse JSON into jq value.
    - tojson/tojson($opts) Serialize jq value into JSON. $opts are:
        - {indent: number} Indent depth.
    - from_jsonl Parse JSON lines into jq array.
    - to_jsonl Serialize jq array into JSONL.

  jq-flavoured JSON

    - from_jq Parse jq-flavoured JSON into jq value.
    - to_jq/to_jq($opts) Serialize jq value into jq-flavoured JSON. jq-flavoured JSON has optional key quotes, # comments and can have trailing comma in objects. $opts are:
        - {indent: number} Indent depth.

Note that fromjson and tojson use different naming conventions as they originate from jq’s standard library.

  YAML

    - from_yaml Parse YAML into jq value.
    - to_yaml/to_yaml($opts) Serialize jq value into YAML. $opts are:
        - {indent: number} Indent depth.

  TOML

    - from_toml Parse TOML into jq value.
    - to_toml/to_toml($opts) Serialize jq value into TOML. $opts are:
        - {indent: number} Indent depth.

  CSV

    - from_csv/from_csv($opts) Parse CSV into jq value.
    To work with tab separated values you can use from_csv({comma: "\t"}) or fq -d csv -o 'comma="\t"'.
    $opts are:
        - {comma: string} field separator, default ",".
        - {comment: string} comment line character, default "#".
    - to_csv/to_csv($opts) Serialize jq value into CSV. $opts are:
        - {comma: string} field separator, default ",".

  XML entities

    - from_xmlentities Decode XML entities.
    - to_xmlentities Encode XML entities.

  URL

    - from_urlpath Decode URL path component.
    - to_urlpath Encode URL path component. Whitespace as %20.
    - from_urlencode Decode URL query encoding.
    - to_urlencode Encode URL to query encoding. Whitespace as "+".
    - from_urlquery Decode URL query into object. For duplicate keys value will be an array.
    - to_urlquery Encode object into query string.
    - from_url Decode URL into object.

        > "schema://user:pass@host/path?key=value#fragment" | from_url
        {
          "fragment": "fragment",
          "host": "host",
          "path": "/path",
          "query": {
            "key": "value"
          },
          "rawquery": "key=value",
          "scheme": "schema",
          "user": {
            "password": "pass",
            "username": "user"
          }
        }
    - to_url Encode object into URL string.

  PEM

    - from_pem Decode PEM to binary.
    - to_pem/to_pem($label) Encode binary to PEM. $label is the PEM header label.

  Hex and base64

    - from_hex Decode hex string to binary.
    - to_hex Encode binary into hex string.
    - from_base64/from_base64($opts) Decode base64 encodings into binary. $opts are:
    - {encoding:string} encoding variant: std (default), url, rawstd or rawurl
    - to_base64/to_base64($opts) Encode binary into base64 encodings. $opts are:
    - {encoding:string} encoding variant: std (default), url, rawstd or rawurl

  Hash functions

    - to_md4 Hash binary using md4.
    - to_md5 Hash binary using md5.
    - to_sha1 Hash binary using sha1.
    - to_sha256 Hash binary using sha256.
    - to_sha512 Hash binary using sha512.
    - to_sha3_224 Hash binary using sha3 224.
    - to_sha3_256 Hash binary using sha3 256.
    - to_sha3_384 Hash binary using sha3 384.
    - to_sha3_512 Hash binary using sha3 512.

  Text encodings

    - to_iso8859_1 Encode string as ISO8859-1 into binary.
    - from_iso8859_1 Decode binary as ISO8859-1 into string.
    - to_utf8 Encode string as UTF8 into binary.
    - from_utf8 Decode binary as UTF8 into string.
    - to_utf16 Encode string as UTF16 into binary.
    - from_utf16 Decode binary as UTF16 into string.
    - to_utf16le Encode string as UTF16 little-endian into binary.
    - from_utf16le Decode binary as UTF16 little-endian into string.
    - to_utf16be Encode string as UTF16 big-endian into binary.
    - from_utf16be Decode binary as UTF16 big-endian into string.

INTERACTIVE REPL

The REPL can be useful in some scenarios:

    - When decoding is slow you can reuse the decode result.
    - Dig thru a file using sub-REPL to cut down on typing.
    - Use auto-completion to speed up typing.

    # start REPL with no (null) input
    $ fq -i
    null>
    # same as
    $ fq -ni
    null>
    # in the REPL you will see a prompt indicating current input and you can type a jq expression to evaluate.
    # start REPL with one file as input
    $ fq -i . doc/file.mp3
    mp3>
    # basic arithmetic and jq expressions
    mp3> 1+1
    2
    mp3> 1, 2, 3 | . * 2
    2
    4
    6
    mp3> [1, 2, 3] | add
    6
    # "." is the identity function which just returns current input, the mp3 file.
    mp3> .
    # access the first frame in the mp3 file
    mp3> .frames[0]
    # start a new nested REPL with first frame as input
    mp3> .frames[0] | repl
    # prompt shows "path" to current input and that it's an mp3_frame.
    # Ctrl-D to exit REPL or to shell if last REPL
    > .frames[0] mp3_frame> ^D
    # "jq" value of layer in first frame
    mp3> .frames[0].header.layer | tovalue
    3
    mp3> .frames[0].header.layer * 2
    6
    # symbolic value, same as "jq" value
    mp3> .frames[0].header.layer | tosym
    3
    # actual underlying decoded value
    mp3> .frames[0].header.layer | toactual
    1
    # description of value
    mp3> .frames[0].header.layer | todescription
    "MPEG Layer 3"
    mp3> ^D
    $ # back to shell

Use Ctrl-D to exit and Ctrl-C to interrupt current evaluation.

EXAMPLES

  Basic usage

fq tries to behave the same way as jq as much as possible, so you can do:

    fq . file
    fq < file
    cat file | fq
    fq . < file
    fq . *.png *.mp3
    fq '.frames[0]' *.mp3
    fq '.frames[-1] | tobytes' file.mp3 > last_frame

  Common usages

    # recursively display decode tree but truncate long arrays
    fq d file
    # same as
    fq display file
    # display all bytes for each value
    fq dd file
    # same as
    fq 'd({array_truncate: 0, string_truncate: 0, display_bytes: 0})' file
    # display 200 bytes for each value
    fq 'd({display_bytes: 200})' file
    # recursively display decode tree without truncating
    fq da file
    # same as
    fq 'd({array_truncate: 0, string_truncate: 0})' file
    # display a specific decode tree one level
    fq '.path[1].to.value' file
    # display a specific decode tree all levels
    fq '.path[1].to.value | d' file
    fq '.path[1].to.value | dd' file
    fq '.path[1].to.value | da' file
    # recursively and verbosely display decode tree
    fq dv file
    # same as
    fq 'd({array_truncate: 0, string_truncate: 0, verbose: true})' file
    # JSON representation for whole file
    fq tovalue file
    # or use -V (--value-output) that does tovalue automatically
    fq -V . file
    # or -Vr if the value is a string and you want a "raw" string
    fq -Vr .path.to.string file
    # JSON but raw bit fields truncated
    fq -o bits_format=truncate tovalue file
    # JSON but raw bit fields as md5 hex string
    fq -o bits_format=md5 tovalue file
    # JSON but raw bit fields as byte arrays
    fq -o bits_format=byte_array tovalue file
    # look up a path
    fq '.some[1].path' file
    # look up a path and output JSON
    fq -V '.some[1].path' file
    # can be a query that outputs multiple values
    # this outputs first and last value in .some array and .path, three values in total
    fq -V '.some[0,-1], .path' file
    # grep whole tree by value
    fq 'grep("^prefix")' file
    fq 'grep(123)' file
    # grep whole tree by condition
    fq 'grep_by(. >= 100 and . <= 200)' file
    # recursively look for values fulfilling some condition
    fq '.. | select(.type=="trak")?' file
    fq 'grep_by(.type=="trak")' file
    # grep_by(f) is an alias for .. | select(f)?, that is: recurse, select and ignore errors
    # recursively look for decode value roots for a format
    fq '.. | select(format=="jpeg")' file
    # can also use grep_by
    fq 'grep_by(format=="jpeg")' file
    # recursively look for first decode value root for a format
    fq 'first(.. | select(format=="jpeg"))' file
    fq 'first(grep_by(format=="jpeg"))' file
    # decode file as mp4 and return a result even if there are some errors
    fq -d mp4 file.mp4
    # decode file as mp4 and also ignore validity assertions
    fq -o force=true -d mp4 file.mp4

LINKS

COPYRIGHT

See LICENSE file in source tree.

SEE ALSO

jq(1)

AUTHOR

Mattias Wadman