CHANGES.md
fromjson now works more like jq and as one would expect for decode value. fromjson used to be implement as normal format decode which returns decode values which in turn when decoding from will decode the backing binary range. This result was expressions like "123" | fromjson | fromjson would return 123 instead of failing the second fromjson. #1268 Thanks @soberich for reportingiprint now support strings in addition to numbers #1246
$ fq -cn '"fq" | iprint'
{"bin":"0b1100110","dec":"102","hex":"0x66","oct":"0o146","str":"f"}
{"bin":"0b1110001","dec":"113","hex":"0x71","oct":"0o161","str":"q"}
input_filename used to always return the last filename. Now it returns null which is wrong but less confusing. The REPL mode needs quite a bit of refactor to fix this properly. #1251avc_sei Decode pic_timing and buffering_period. #1271bplist Properly decode unicode strings. #1270 Thanks @soberich for reportingicc_profile Decode curve type #12651 // 2 as $v | $vavc_nal Decode slice header frame number and picture order #1227flac_frame Add option to decode sample and residuals details #802matroska Update to latest spec #1219mpeg_asc Decode SBR and PS hierarchical signalling #1192 Thanks @ntrrgcmp4 Decode elng box #1182safetensors Add decoder #1203 Thanks @LeowbattleMinor release to maninly fix golang sum database hash issue.
go get and go install to build a binary that displays the incorrect fq version, and if GOPROXY=direct is used it would cause a security error.avc_nalu Decode payload into a rbsp struct (raw byte sequence payload) and only introduce a sub-buffer if de-emulation prevention was done. #1120hevc_nalu Decode payload into a rbsp struct. See above. #1120matroska Update specification. #1124mp4
skip_samples option to not add fields or decode samples at all. Useful when dealing with broken mp4 files. #1089truncated_size field if allow_truncated is used and we had to truncate. Useful when dealing with broken mp4 files. #1091pcmC Decode box. #1082chnl Decode box. #1082ipcm track format. Could end up with a huge array and not be very useful. #1083trun Skip samples array if there are no sample flagsuuid Decode tfxd and tfrf boxes. #1110 Thanks @rafalborczuchdebug/0/debug/1 (correct prefix) and stderr/0 (output raw strings). #1015 #1016av1_obu Decode more fields and derive more values according to spec. #1020 #1021elf Handle section header null better. #1051markdown Update to latest gomarkdown (Fixes CVE-2024-44337). #1048matroska Spec update. #1035midi - Mapped SMPTE frame rates to strings and fixed bug in SMPTE offset metaevent decoding and more refactoring. #1023 Thanks @transcriptazetoml to_toml/1 now support indent option. #1033wav Decode fmt chunk a bit better and improve format sym names (implied endian not bit size). #1034 #1038yaml to_yaml/1 now support indent option. #1033golang.org/x/exp dependencyexp/slices package with standard librarymidi, negentropy, tap and txz, see below for details.byte_array bits format. #992
# by default binary values are UTF-8 strings
$ fq -V -n '[1,2,3] | tobytes'
# with -o bits_format=byte_array they are byte arrays instead
"\u0001\u0002\u0003"
$ fq -cV -n -o bits_format=byte_array '[1,2,3] | tobytes'
[1,2,3]
# data is a string
$ fq '.program_headers[] | select(.type=="interp")' /bin/ls
{"align":1,"data":"/lib/ld-musl-x86_64.so.1\u0000", ...}
# data is a byte array
$ fq -cV -o bits_format=byte_array '.program_headers[] | select(.type=="interp")' /bin/ls
{"align":1,"data":[47,108,105,98,47,108,100,45,109,117,115,108,45,120,56,54,95,54,52,46,115,111,46,49,0], ...}
add/1 functionmatroska Updated to latest specification. #1010midi MIDI decoder added by @twystd. #1004negentropy Negentropy message decoder added by @fiatjaf. #1012tap and txz TAP and TXZ tape format for ZX Spectrum computers added by @mrcook. #975REPL word navigation fix and jpeg DHT decoding, otherwise mostly maintenance release to update dependencies.
jpeg
New iNES/NES 2.0 ROM decoder (thanks @mlofjard) and basic JPEG 2000 format support. jq language improvements and fixes from gojq. And as always various decoder improvements and fixes.
string_truncate option to configure how to truncate long strings when displaying a decode value tree. dd, dv etc set truncate length to zero to not truncate. #919ltrim, rtrim, and trim functions{x:0,y:1} | {a:.x,a:.y})halt and halt_error functions to stop the command execution immediately"a" as $v | def f: $v; "b" as $v | f)ltrimstr and rtrimstr functions to emit error on non-string inputnearbyint and rint functions to round ties to evenreduce, foreach, if, try-catch syntax as object valuespow10 in favor of exp10, define scalbn and scalbln by ldexpltrimstr/rtrimstr.fit
jp2c New JPEG 2000 codestream decoder. #928icc_profile Strip whitespace in header and tag strings. #912mp4
jp2c, jp2h, ihdr jP JPEG 2000 related boxes support. #928thmb box support. #897hdlr component name might be zero bytes. #896nes New iNES/NES 2.0 ROM decoder (thanks @mlofjard). #893Adds support for various LevelDB formats (thanks @mikez) and Garmin Flexible and Interoperable Data Transfer format (FIT) (thanks @mlofjard).
And as usual some small fixes and dependency updates.
~/.config/fq in addition to ~/Library/Application Support/fq. #871aac_frame Decode instance tag and common window flag. #859
fit Add support for Garmin Flexible and Interoperable Data Transfer decoder. Thanks @mlofjard #874
# to_kml.jq
# convert locations in a fit structure to KML
def to_kml:
( [ .data_records[].data_message as {position_lat: $lat, position_long: $long}
| select($lat and $long)
| [$long, $lat, 0]
| join(",")
]
| join(". ")
| { kml: {"@xmlns":"http://earth.google.com/kml/2.0"
, Document: {Placemark: {LineString: {coordinates: .}}}}
}
| to_xml({indent:2})
);
Usage:
# -L to add current directory to library path
# -r for raw string output
# 'include "to_ml";' to include to_kml.jq
# to_kml calls to_kml function
$ fq -L . -r 'include "to_kml"; to_kml' file.fit > file.kml
hevc_sps Fix some incorrect profile_tier_level decoding. #829
html Fix issue parsing elements including SOLIDUS "/". #870
mpeg_es Support ES_ID_Inc and decode descriptors for IOD tags
leveldb_descriptor, leveldb_log, leveldb_table Add support for LevelDB. Thanks @mikez #824
pcapng Decode all section headers instead of just the first. #860
png Fix incorrect decoding of type flags. #847
hevc_sps Fix incorrect decoding of profile_tier_level. #829
tls Fix field name typos. #839
mp4
ctts box v0 sample offset seems to be signed in practice but not in spec. #832webp Decode width, height and flags for lossless WebP. #857
torepr for descriptorstringify by preallocating result<byte>[.<bits>] notation it is now shown as 5-6, before it was shown as 5-5.7. See usage documentation for more examples. #789from_jq error handling. Thanks @emanuele6 for reporting. #788pick/1 function. Outputs value and nested values for given input expression.
# pick out all string values from a ELF header
$ fq '.header | pick(.. | select(type == "string"))' /bin/ls
{
"ident": {
"data": "little_endian",
"magic": "\u007fELF",
"os_abi": "sysv",
"pad": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
},
"machine": "x86_64",
"type": "dyn"
}
# bonus tip:
# if you only need to pick one level then jq has a shortcut syntax
$ fq '.header | {type, machine}' /bin/ls
{
"machine": "x86_64",
"type": "dyn"
}
avi
decode_extended_chunks option. This is trailing chunks used by big AVI files. #786$ fq '.streams[0]' file.avi
│00 01 02 03 04 05 06 07│01234567│.streams[0]{}: stream
0x1680│ 00 00 00 01 67 f4│ ....g.│ samples[0:3]:
0x1688│00 0d 91 9b 28 28 3f 60│....((?`│
0x1690│22 00 00 03 00 02 00 00│".......│
* │until 0x2409.7 (3464) │ │
│ │ │ type: "vids"
│ │ │ handler: "H264"
│ │ │ compression: "H264"
exif (and tiff)
gzip
members array and a uncompressed field that is the concatenation of the uncompressed members.macho
matroska
protobuf
webp
zip Better timestamp support and fixes
$ fq '.local_files[] | select(.file_name == "a").last_modification' file.zip
│00 01 02 03 04 05 06 07│01234567│.local_files[3].last_modification{}:
0xd0│ 81 01 │ .. │ fat_time: 0x181
│ │ │ second: 2 (1)
│ │ │ minute: 12
│ │ │ hour: 0
0xd0│ 73 53│ sS│ fat_date: 0x5373
│ │ │ day: 19
│ │ │ month: 11
│ │ │ year: 2021 (41)
│ │ │ unix_guess: 1637280722 (2021-11-19T00:12:02)
Fix handling of shadowing order for _<name> keys, 3 new decoders caff, moc3 and opentimestamps, some smaller fixes and improvements.
In other jq news jq 1.7 was finally released 5 years since the last release! also happy to now be part of the jq maintainance team.
caff and moc3. See below for details._<name>) for text formats like json, yaml etc. #757
This happenned because fq has a bunch of internal underscore prefixed "extra" keys that is used for various things and these had priority even when there already existed a "value" key with same name.
$ fq -n '`{"_format": 123}` | fromjson | ._format'
123, before "json".
$ fq -n '`{}` | fromjson | ._missing'
null, before error--null/nul-output to --raw-output0 and also clarify that NUL and new lines are outputted after and not between each output.
This is to be in sync with jq (https://github.com/jqlang/jq/pull/2684). #736abs functionleaf_paths functionsplit in combination with binary to not include separator. #767caff Add archive format decoder. Thanks @Ronsor #747
.cmo3 and .can3 used by Live2D Cubism.id3v2 Handle W000-WZZZ and W00-WZZ URL frames. #758matroska Update spec and regenerate. #737moc3 Add Live2D Cubism MOC3 decoder. Thanks @Ronsor #747
mp3_frame_xing Detect lame ext more similar to ffmpeg and mediainfo. #763mp4
sgpd (Sample group definition box) entries. Thanks Sergei Kuzmin @ksa-real #707cslg (Composition to decode timeline mapping) box. #754emsg (Event message) and id3v2 message data. #755ftyp. #723opentimestamps Add OpenTimestamps decoder. Thanks @fiatjaf #769Added LuaJIT bytecode decoder by @dlatchx, otherwise mostly small things. Been busy with nice weather and helping our getting jq development and maintenance back on track.
tovalue on binary, in addition decode value binary, respect bits_format options. #677
# uses -V to do tovalue to get a hex string
# uses -r to output "raw" string
$ fq -o bits_format=hex -Vr -n '[1,0xff,3] | tobytes'
01ff03
@urid URI format functionluajit Add LuaJIT bytecode decoder. Thanks @dlatchx #709mp4 Improved sample group definition box sgpd entries decoder. Thanks @ksa-real #707mp4 Improved user metadata udta structure decoding #676wav Decode bext chunk. #712Adds decoders for PostgreSQL btree, control and heap files. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka
Adds new option skip gaps and output binary as hex string.
Make bits/bytes formats work a bit more intuitive.
Bug fixes to to_hex, to_base64 and trim functions.
Bug fixes and additions to bson, bitcoin_transaction, mp4, id3v2, html, and matroska formats.
bits,bytes now are real binaries and not raw decode value. This means they behave more like you would expect. #666
# build your own strings(1)-like tool:
# scan matches range in a binary using a regexp and output ranges as new binaries
# \w\s looks for whitespace and alpha/numeric characters
# uses `...` raw string literal to not have to escape
# select/test to only include strings containing "thread"
# dd to display with no truncation
$ fq -d bytes 'scan(`[\w\s]{16,}`) | select(test("thread")) | dd' file.mp4
│00 01 02 03 04 05 06 07│01234567│
0x250│36 20 6c 6f 6f 6b 61 68│6 lookah│.: raw bits 0x250-0x262.7 (19)
0x258│65 61 64 5f 74 68 72 65│ead_thre│
0x260│61 64 73 │ads │
│00 01 02 03 04 05 06 07│01234567│
0x260│ 31 20 73 6c│ 1 sl│.: raw bits 0x264-0x273.7 (16)
0x268│69 63 65 64 5f 74 68 72│iced_thr│
0x270│65 61 64 73 │eads │
to_hex,to_base64 now correctly handles raw decode values, before the raw bits would be turned into codepoints and then binary UTF-8 possibly introducing invalid codepoints (0xfffd). Thanks @Rogach #672
$ fq -r '.uncompressed | to_hex' file.gz
f6f2074cf77d449d
# with the change to add hex bits format you can also do this now
$ fq -Vr -o bits_format=hex .uncompressed file.gz
f6f2074cf77d449d
tovalue Now output a "deep" jq value, before it was shallowly a jq value which could be confusing, ex tovalue | .header could be a decode value.skip_gaps for -V/tovalue can be used to filter out gap fields when represented as JSON. Gaps are bit ranges that no decoder added any field for. #649
gap0 etc.bits_format=hex to represent raw bits as hex string in JSON. #673
# output decode tree, JSON with binaries as strings and JSON with binaries as hex strings
$ fq '.packets[0].packet | ., tovalue, tovalue({bits_format:"hex"})' file.pcap
│00 01 02 03 04 05 06 07│01234567│.packets[0].packet{}: (ether8023_frame)
0x28│8c 85 90 74 b8 3b │...t.; │ destination: "8c:85:90:74:b8:3b" (0x8c859074b83b)
0x28│ e8 de│ ..│ source: "e8:de:27:c8:9a:6e" (0xe8de27c89a6e)
0x30│27 c8 9a 6e │'..n │
0x30│ 08 06 │ .. │ ether_type: "arp" (0x806) (Address Resolution Protocol)
0x30│ 00 01│ ..│ payload: raw bits
0x38│08 00 06 04 00 01 e8 de│........│
0x40│27 c8 9a 6e c0 a8 01 01│'..n....│
0x48│00 00 00 00 00 00 c0 a8│........│
0x50│01 e6 │.. │
{
"destination": "8c:85:90:74:b8:3b",
"ether_type": "arp",
"payload": "\u0000\u0001\b\u0000\u0006\u0004\u0000\u0001\ufffd\ufffd'Țn\ufffd\ufffd\u0001\u0001\u0000\u0000\u0000\u0000\u0000\u0000\ufffd\ufffd\u0001\ufffd",
"source": "e8:de:27:c8:9a:6e"
}
{
"destination": "8c:85:90:74:b8:3b",
"ether_type": "arp",
"payload": "0001080006040001e8de27c89a6ec0a80101000000000000c0a801e6",
"source": "e8:de:27:c8:9a:6e"
}
d/display on a binary will now always hexdump, this feels more intuitive. Before it could output raw binary as display is used as an implicit output function. Implicit (you don't mention d at all) can still output raw binary. #665
# outputs raw binary (if stdout is not a tty)
$ fq -n '[1,2,3] | tobytes' | xxd
00000000: 0102 03
# outputs hexdump (even if stdout is not a tty) as we explicitly use d
$ fq -n '[1,2,3] | tobytes | d' | cat
|00 01 02 03 04 05 06 07|01234567|
0x0|01 02 03| |...| |.: raw bits 0x0-0x2.7 (3)
trim can now handle multi-line strings. #668bits,bytes Is a proper binary not a raw decode value. #666bitcoin_transaction Properly decode witness items array. #671 Thanks @Rogachbson Add javascript, decimal128, minkey and maxkey support. Fix decoding of datetime and use the correct size type for binary and document size. Thanks Matt Dale @matthewdale. #650id3v2
html Is now probeable. #667matroska Fallback raw is probe fail or file_data #645mp4
ctts,infe,iinf,trun more proper decoding based on version. #643pg_btree,pg_control,pg_heap Add PostgreSQL btree, control and heap support. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka. #415Mostly a bug fix release but adds -V for easy JSON output.
Add -V argument to default output JSON instead of decode tree in case of decode value. #385 Thanks @peterwaller-arm for reminding me to merge this.
# default in case of decode value is to show a hexdump tree
$ fq '.headers | grep_by(.id=="TSSE").text' file.mp3
│00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
0x0c│ 4c 61 76│ Lav│.headers[0].frames[0].text: "Lavf58.76.100"
0x18│66 35 38 2e 37 36 2e 31 30 30 00 │f58.76.100. │
# with -V an implicit "tovalue" is done
$ fq -V '.headers | grep_by(.id=="TSSE").text' file.mp3
"Lavf58.76.100"
# and in combination with -r will for strings output a "raw string" without quotes
# for other types like number, object, array etc -r makes not difference (same as jq)
$ fq -Vr '.headers | grep_by(.id=="TSSE").text' file.mp3
Lavf58.76.100
As a side note -V can be used with binary type also. Then the binary data will be interpreted as UTF-8 and turned into a string.
# trailing null terminator ends up as codepoint zero `\u0000`
$ fq -V '.headers | grep_by(.id=="TSSE").text | tobytes' file.mp3
"Lavf58.76.100\u0000"
# with -r null terminator and a new line is outputted
$ fq -Vr '.headers | grep_by(.id=="TSSE").text | tobytes' file.mp3 | hexdump -C
00000000 4c 61 76 66 35 38 2e 37 36 2e 31 30 30 00 0a |Lavf58.76.100..|
0000000f
# in contrast raw binary output has no new line separator
$ fq '.headers | grep_by(.id=="TSSE").text | tobytes' doc/file.mp3 | hexdump -C
00000000 4c 61 76 66 35 38 2e 37 36 2e 31 30 30 00 |Lavf58.76.100.|
0000000e
Fix issue using decode value in object passed as argument to internal function. #638
# this used to fail but now works
fq '.tracks[0].samples[10] | avc_au({length_size: <decode value>})' file.mp4
Some typo fixes. Thanks @retokromer and @peterwaller-arm
aiff Basic AIFF decoder added. #614matroska Update to latest specification. #640msgpack Fix bug decoding some fixstr lengths. #636 Thanks @schmee for reporting.TLS decode and decryption, better streaming matroska/webm support, support raw IP in PCAP and bug fixes.
-o name=value) to nested decoders. #589
-o name=@path fails to read file at path. #597id3v2 Properly decode CTOC subframes. #606matroska
decode_samples option. #574pcap,pcapng Support raw IPv4 and IPv6 link frames. #599 #590tls Add Transport layer security decoder and decryption. #603
# show first 50 bytes of decrypted client/server TLS application data stream
# -o [email protected] is used to read keylog from a file
# first .stream is TCP stream, second .stream the application data stream
$ fq -o [email protected] '.tcp_connections[0].["client", "server"].stream.stream | tobytes[0:50] | dd' file.pcap
│00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
0x00│47 45 54 20 2f 64 75 6d 70 2f 6c 6f│GET /dump/lo│.: raw bits 0x0-0x31.7 (50)
0x0c│67 20 48 54 54 50 2f 31 2e 31 0d 0a│g HTTP/1.1..│
0x18│48 6f 73 74 3a 20 69 6e 77 61 64 65│Host: inwade│
0x24│72 2e 63 6f 6d 0d 0a 55 73 65 72 2d│r.com..User-│
0x30│41 67 │Ag │
│00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
0x00│48 54 54 50 2f 31 2e 31 20 32 30 30│HTTP/1.1 200│.: raw bits 0x0-0x31.7 (50)
0x0c│20 4f 4b 0d 0a 41 63 63 65 70 74 2d│ OK..Accept-│
0x18│52 61 6e 67 65 73 3a 20 62 79 74 65│Ranges: byte│
0x24│73 0d 0a 43 6f 6e 74 65 6e 74 2d 4c│s..Content-L│
0x30│65 6e │en │
# show first TLS record from server
$ fq '.tcp_connections[0].server.stream.records[0] | d' file.pcap
│00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│.tcp_connections[1].server.stream.records[0]{}: record
0x00│16 │. │ type: "handshake" (22) (valid)
0x00│ 03 03 │ .. │ version: "tls1.2" (0x303) (valid)
0x00│ 00 40 │ .@ │ length: 64
│ │ │ message{}:
0x00│ 02 │ . │ type: "server_hello" (2)
0x00│ 00 00 3c │ ..< │ length: 60
0x00│ 03 03 │ .. │ version: "tls1.2" (0x303)
│ │ │ random{}:
0x00│ 86│ .│ gmt_unix_time: 2249760024 (2041-04-16T21:20:24Z)
0x0c│18 9d 18 │... │
0x0c│ 19 92 33 c2 21 ce 4f 97 30│ ..3.!.O.0│ random_bytes: raw bits
0x18│28 98 b3 fd 1e 15 f4 36 bb e9 14 f4│(......6....│
0x24│67 61 66 79 d5 3f 06 │gafy.?. │
0x24│ 00 │ . │ session_id_length: 0
│ │ │ session_id: raw bits
0x24│ c0 2f │ ./ │ cipher_suit: "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" (0xc02f)
0x24│ 00 │ . │ compression_method: "null" (0x0)
0x24│ 00│ .│ extensions_length: 20
0x30│14 │. │
│ │ │ extensions[0:2]:
│ │ │ [0]{}: extension
0x30│ ff 01 │ .. │ type: "renegotiation_info" (65281)
0x30│ 00 01 │ .. │ length: 1
0x30│ 00 │ . │ data: raw bits
│ │ │ [1]{}: extension
0x30│ 00 10 │ .. │ type: "application_layer_protocol_negotiation" (16)
0x30│ 00 0b │ .. │ length: 11
0x30│ 00 09│ ..│ serer_names_length: 9
│ │ │ protocols[0:1]:
│ │ │ [0]{}: protocol
0x3c│08 │. │ length: 8
0x3c│ 68 74 74 70 2f 31 2e 31 │ http/1.1 │ name: "http/1.1"
# use ja3.jq to calculate ja3 TLS fingerprint
# https://github.com/wader/fq/blob/master/format/tls/testdata/ja3.jq
$ fq -L path/to/ja3 'include "ja3"; pcap_ja3' file.pcap
[
{
"client_ip": "192.168.1.193",
"client_port": 64126,
"ja3": "771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-16-22-23-49-13-43-45-51-21,29-23-30-25-24,0-1-2",
"ja3_digest": "bc29aa426fc99c0be1b9be941869f88a",
"server_ip": "46.101.135.150",
"server_port": 443
}
]
toml Fail faster to speed up probe. Could in some cases read the whole file before failing. Thanks @0-wiz-0 for report. #594zip Properly decode EOCD record in zip64 files. Thanks @0-wiz-0 for report and spec interpretation. #586 #596xml Fail faster to speed up probe. Could in some cases read the whole file before failing. Thanks @0-wiz-0 for report. #594Bug fix release, no new features mostly due to holidays and busy with other things (some jq related!).
Also been preparing for a talk about fq this weekend at FOSDEM 2023.
mp3 Add option for max unknown bits to handle more mis-probing. Default to 50%mp4
ftyp set minor description to date for "qt" filestkhd decode enabled, preview, etc flagsudta Handle case with box type is key and value rest of boxsgpd,sbgp Change grouping type to a string as it seems to be what it is in practice.tcp_segment Decode all standard options, MSS, Window scale, timestamp etc. Rename "maxseg" to "mss".This ended up being a release to cleanup old sins in the decoder internals and change some defaults how binary values work with JSON and string functions.
It also adds a new Time Zone Information Format decoder tzif (Thanks Takashi Oguma @bitbears-dev) and a new Apple BookmarkData decoder apple_bookmark decoder (Thanks David McDonald @dgmcdona). Also a new function from_ns_keyed_archiver was added to convert NSKeyedArchiver encoded objects into JSON.
A possible breaking change is that now all from/to prefix functions now has a from_/to_ prefix, ex: from_mp3 instead of frommp3. There are some few exceptions to this. Note that the functions named just be the format name, ex mp3 are still around.
In other fq related news jq-lsp got some fixed and additions and seems to work fine with neovim. It's also possible to use jq-lsp with vscode using vscode-jq.
from/to prefix now has the prefix from_/to_. This is to be easier to read and more consistent, there are still some exceptions like tovalue, torepr, tobytes etc but in general anything that does not deal with primitive types is now snake_case. #535bit_formats option value (how raw bits values are represented in JSON) from snippet to string. snippet meant truncated bits as base64. Now all bits are included as a UTF-8 string. The string will be binary safe (not lose any data) when used internally in fq but will lose data when represented in JSON as some bytes can't be encoded as UTF-8. #499tobytes etc to convert to binary value. #540
# This used to not work as test/1 would convert decode values to the source bytes
# (0x00 0x00 0x00 0x01) in this case. Now the jq value (symbolic in this case) will
# be used instead. You can do ".test | tobytes" to get old behavior.
#
# find all types with a "mdta." prefix
$ fq -o line_bytes=10 'grep_by(.type | test(`^mdta\.`))' file.mp4
│00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[0]{}: box
0x528│ 00 00 00 1c │ .... │ size: 28
0x528│ 00 00 00 01│ ....│ type: "mdta.title" ("\x00\x00\x00\x01")
0x532│00 00 00 14 64 61 74 61 00 00│....data..│ boxes[0:1]:
0x53c│00 01 00 00 00 00 74 65 73 74│......test│
│00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[1]{}: box
0x546│00 00 00 25 │...% │ size: 37
0x546│ 00 00 00 02 │ .... │ type: "mdta.encoder" ("\x00\x00\x00\x02")
0x546│ 00 00│ ..│ boxes[0:1]:
0x550│00 1d 64 61 74 61 00 00 00 01│..data....│
0x55a│00 00 00 00 4c 61 76 66 35 39│....Lavf59│
0x564│2e 32 37 2e 31 30 30│ │.27.100│ │
.[<JQValue here>] = 123 #509unknown# to gap#. "unknown" is probably a useful field name in some formats and "gap" describe better what it is. #500apple_bookmark New Apple BookmarkData decoder. Thanks David McDonald @dgmcdona. #493bplist
lost_and_found array with unused valuesfrom_ns_keyed_archiver, see bplist docs for details on how to use it. Thanks David McDonald @dgmcdona. #493# decode bplist, from_ns_keyed_archiver converts to JSON plist and then into object data as JSON, find app bookmarks keys and expand them as bookmark data and convert to represented JSON, and finally build path to applications
$ fq -r 'from_ns_keyed_archiver | (.. | .Bookmark? // empty) |= (apple_bookmark | torepr) | .. | .target_path? // empty | join("/")' recentapps.sfl2
System/Applications/Utilities/Terminal.app
Applications/Spotify.app
System/Applications/Calculator.app
System/Applications/Preview.app
Applications/Alacritty.app
Applications/DB Browser for SQLite.app
System/Applications/System Preferences.app
System/Library/CoreServices/Applications/Directory Utility.app
System/Applications/Utilities/Activity Monitor.app
Applications/Safari.app
tzif new Time Zone Information Format decoder. Thanks Takashi Oguma @bitbears-dev. #498mp4
mdta metadata namespace and key names for ilst child boxes. #521$ fq 'grep_by(.type=="ilst").boxes | map({key: .type, value: .boxes[0].data}) | from_entries' file.mp4
# create object with all ilst key/value pairs
{
"mdta.encoder": "Lavf59.27.100",
"mdta.title": "test"
}
# query specific value
$ fq -r 'grep_by(.type=="mdta.encoder").boxes[0].data | tovalue' file.mp4
Lavf59.27.100
sidx version 1. #506# which boxes has depends_on flags
$ fq 'grep_by(.sample_depends_on) | parent.type' fragmented.mp4
pdin boxes. #524hnti boxes. #513mp3_tags Add VBRI support and split into into mp3_frame_xing and mp3_frame_vbri decoders. #525Adds avi decoder and replace raw with more convenient bits and bytes format. Otherwise mostly small updates and bug fixes.
Increase minor version. fq does not use semantic versioning (yet) but it's probably a good idea to start increase minor version when adding features to be able to do patch releases.
In other fq related news:
raw format with bits and bytes format that decodes directly to a binary with different unit size.
$ echo -n 'hello' | fq -d bytes '.[-3:]' > last_3_bytes
$ echo 'hello' | fq -d bytes '.[1]'
101
$ echo 'hello' | fq -c -d bits '[.[range(8)]]'
[0,1,1,0,1,0,0,0]
avc_au Support annexb format (used in AVI). #476avi Add AVI (Audio Video Interleaved) decoder. #476
# extract samples for stream 1
$ fq '.streams[1].samples[] | tobytes' file.avi > stream01.mp3
bits Replaces raw but is a binary using bit units. #485bytes Replaces raw but is a binary using byte units. #485bplist
flac_frame Correctly decode zero escape sample size. #461id3v2 Fix decoding of COMM and TXXX with missing null terminator. #468matroska Updated to latest specification. #455mp3_frame Use frame size calculation from spec instead of own as it seems to not work in some cases. #480mp3_frame_tags Replaces xing and also decodes "lame extensions" for both Xing and Info. #481raw Removed. #485wav More codec symbol names and now shares RIFF code with AVI decoder. #476yaml Fix type panic for large integers. #462bplist Apple Binary Property List decoder. Thanks David McDonald @dgmcdona #427markdown decoder. #422fq tovalue file etc), #440debug (and some other native go iterator functions) inside path(...), which is used by assign (... = ...) expressions etc. #439toactual and tosym work more similar to tovalue. #432# or help(matroska) in the REPL
$ fq -h matroska
matroska: Matroska file decoder
Decode examples
===============
# Decode file as matroska
$ fq -d matroska . file
# Decode value as matroska
... | matroska
Lookup element using path
=========================
$ fq 'matroska_path(".Segment.Tracks[0)")' file.mkv
Get path to element
===================
$ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv
References
==========
- https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
- https://matroska.org/technical/specs/index.html
- https://www.matroska.org/technical/basics.html
- https://www.matroska.org/technical/codec_specs.html
- https://wiki.xiph.org/MatroskaOpus
ar Allow empty integer strings. For example owner id can be an empty string. #428
bitcoin_blkdat Assert that there is a header. As the format is part of the probe group this speeds up probing. #402
bplist Add Apple Binary Property List decoder.
$ fq '.objects.entries[0] | .key, .value' Info.plist
│00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].key{}:
0x32│ 5c │ \ │ type: "ascii_string" (5) (ASCII encoded string)
0x32│ 5c │ \ │ size_bits: 12
│ │ │ size: 12
0x32│ 43 46 42 75│ CFBu│ value: "CFBundleName"
0x3c│6e 64 6c 65 4e 61 6d 65 │ndleName │
│00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].value{}:
0x1ea│ 5f │ _ │ type: "ascii_string" (5) (ASCII encoded string)
0x1ea│ 5f │ _ │ size_bits: 15
0x1ea│ 10 │ . │ large_size_marker: 1 (valid)
0x1ea│ 10 │ . │ exponent: 0
0x1ea│ 18 │ . │ size_bigint: 24
│ │ │ size: 24
0x1ea│ 41 70 70 6c│ Appl│ value: "AppleProResCodecEmbedded"
0x1f4│65 50 72 6f 52 65 73 43 6f 64│eProResCod│
0x1fe│65 63 45 6d 62 65 64 64 65 64│ecEmbedded│
torepr$ fq torepr.CFBundleName Info.plist
"AppleProResCodecEmbedded"
elf
More robust decoding when string tables are missing. #417
# extract entry opcodes and disassemble with ndisasm
$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' \
| ndisasm -b 64 -
00000000 B03C mov al,0x3c
00000002 66BF0600 mov di,0x6
00000006 0F05 syscall
Now decodes program header notes. #421
markdown Add decoder. Is used in fq to render CLI help. #422
# array with all level 1 and 2 headers
$ fq -d markdown '[.. | select(.type=="heading" and .level<=2)?.children[0]]' README.md
[
"fq",
"Usage",
"Presentations",
"Install",
"TODO and ideas",
"Development and adding a new decoder",
"Thanks and related projects",
"License"
]
matroska Add support for sample lacing. Used by FLAC samples etc. #404
mp4 More codec names and also use official names from mp4ra.org. #424
# show details of first two track in file
$ fq -o line_bytes=10 '.tracks[0,1]' big_buck_bunny.mp4
│00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[0]{}: track
0x00910a│20 68 10 01 a0 40 0e 20 8c 1b│ h...@. ..│ samples[0:1295]:
0x009114│c2 2b 99 09 84 42 60 a8 c4 60│.+...B`..`│
* │until 0x541697.7 (5473678) │ │
│ │ │ id: 1
│ │ │ data_format: "mp4a" (MPEG-4 Audio)
│00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[1]{}: track
0x00a5e6│ 00│ .│ samples[0:1440]:
0x00a5f0│00 00 0c 06 00 07 8b 71 b0 00│.......q..│
0x00a5fa│00 03 00 40 80 00 00 00 15 06│...@......│
* │until 0x540959.7 (5464939) │ │
│ │ │ id: 2
│ │ │ data_format: "avc1" (Advanced Video Coding)
html Handle leading doc type and processing directives. #414
New wasm WebAssembly Binary Format decoder by Takashi Oguma @bitbears-dev
# show part of code_section
$ fq '.sections[4].content.code.x[0].code.e | d' add.wasm
│00 01 02 03 04 05 06 07 08 09│0123456789│.sections[4].content.code.x[0].code.e[0:4]:
│ │ │ [0]{}: in
0x3c│ 20│ │ opcode: "local.get" (0x20)
0x46│01 │. │ x: 1 (valid)
│ │ │ [1]{}: in
0x46│ 20 │ │ opcode: "local.get" (0x20)
0x46│ 00 │ . │ x: 0 (valid)
│ │ │ [2]{}: in
0x46│ 6a │ j │ opcode: "i32.add" (0x6a)
│ │ │ [3]{}: in
0x46│ 0b │ . │ opcode: "end" (0xb)
# count opcode usage
$ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' add.wasm
{
"end": 1,
"i32.add": 1,
"local.get": 2
}
# list exports and imports
$ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' add.wasm
{
"export": [
"memory",
"add"
],
"import": []
}
Decode value display now shows address bar on new format or buffer. Should make it easier to spot changes and read hex and ASCII view. #365
Examples of PCAP with different formats and TCP stream buffers:
<pre> ... │<ins>00 01 02 03 04 05 06 07 08 09</ins>│0123456789</ins>│ packet{}: (ether8023_frame) 0x00668│ 00 0a 95 67 49 3c │ ...gI< │ destination: "00:0a:95:67:49:3c" (0xa9567493c) 0x00668│ 00 c0 f0│ ...│ source: "00:c0:f0:2d:4a:a3" (0xc0f02d4aa3) 0x00672│2d 4a a3 │-J. │ 0x00672│ 08 00 │ .. │ ether_type: "ipv4" (0x800) (Internet Prot... │<ins>00 01 02 03 04 05 06 07 08 09</ins>│0123456789</ins>│ payload{}: (ipv4_packet) 0x00672│ 45 │ E │ version: 4 ... │<ins>00 01 02 03 04 05 06 07 08 09</ins>│0123456789</ins>│ payload{}: (tcp_segment) 0x00686│ 00 50 │ .P │ source_port: "http" (80) (World Wide ... ... │ │ │ ipv4_reassembled[0:0]: │ │ │ tcp_connections[0:1]: │ │ │ [0]{}: tcp_connection │ │ │ client{}: │ │ │ ip: "192.168.69.2" │ │ │ port: 34059 │ │ │ has_start: true │ │ │ has_end: true │ │ │ skipped_bytes: 0 │<ins>00 01 02 03 04 05 06 07 08 09</ins>│<ins>0123456789</ins>│ 0x000│47 45 54 20 2f 74 65 73 74 2f│GET /test/│ stream: raw bits 0x000│65 74 68 65 72 65 61 6c 2e 68│ethereal.h│ * │until 0x1bc.7 (end) (445) │ │ ... </pre>Add --unicode-output/-U argument to force use of Unicode characters to improve output readability. #377
fq -cU d file | less -r
to_entries now preserves struct field order. #340
Experimental <code>`raw string`</code> literal support. Work the same as golang raw string literals. Useful in REPL when pasting things etc but should probably be avoided in jq scripts. #371
Properly fail lexing of invalid binary, octal and hex number literals. #371
REPL completion now include all functions. Before some functions with multiple argument counts were skipped. #375
Switch to new gopacket fork with speedup and bug fixes. Remove SLL2 workarounds in fq. #386
csv Correctly handle decode values when tocsv normalize to strings. Before array and object ended up being JSON serialized to strings. #341
$ fq -n '[[1,true,null,"a"]] | tocsv'
"1,true,,a\n"
dns DNS over UDP format was accidentally used to probe TCP streams #366elf Remove redundant program_header structflac
Add 32 bit samples support. #378 Thanks @ktmf01
Properly decode/checksum samples in partitions with raw samples. #379 Thanks @ktmf01
Now successfully decodes all test cases from https://github.com/ietf-wg-cellar/flac-test-files
jsonl Add decoder. Decodes JSON lines. There is also fromjsonl and tojsonl. #374macho
macho_fat format which also fixed handling of file offsets in sections. #362matroska Remove new lines in descriptions. Messes up tree. #369mp3_frame
pcap Support files with nanosecond precision. Has a different magic. #382prores_frame Add basic decoder. Decodes container and fram header. #396 Thanks @Doctor-love for test filestar Fix regression when decoding number fields. Now ok again to be empty string. #372wasm Add WebAssembly Binary Format decoder. #383 Thanks to Takashi Oguma @bitbears-dev
yaml Fail on trailing data. Before it succeeded with the last value. #373zip
xml Correctly handle decode values when toxml normalize to strings. Before array and object ended up being JSON serialized to strings. #341xml
@ instead of - and make it an option attribute_prefix. #401.[1] will be attributes and so on. #357$ fq -nr '{a: {"-boolean": true, "-number": 123, "-null": null}} | toxml'
<a boolean="true" null="" number="123"></a>
<?procinstr?> and improve trailing data error message. #368#seq is found and also properly sort negative #seq. #384make doc-timeout 20m for go test to workaround ci test failgolangci-lint run --fixtoxml, fromxml options for indent, jq mapping variants (object or array) and order preservationfromhtml options for indent, jq mapping variants (object or array) and order preservationtotoml, fromtomltoyaml, fromyamltojq, fromjq options for indent #284
# query a YAML file
$ fq '...' file.yml
# convert YAML to JSON
# note -r for raw string output, without a JSON string with JSON would outputted
$ fq -r 'tojson({indent:2})' file.yml
$ fq -nr '{hello: {world: "test"}} | toyaml, totoml, toxml, tojq({indent: 2})'
hello:
world: test
[hello]
world = "test"
<hello>
<world>test</world>
</hello>
{
hello: {
world: "test"
}
}
$ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a"'
b
$ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a" = "<test>" | toxml({indent: 2})'
<doc>
<element a="<test>"></element>
</doc>
tocsv, fromcsv options for separator and comment character
$ echo -e '1,2\n3,4' | fq -rRs 'fromcsv | . + [["a","b"]] | tocsv'
1,2
3,4
a,b
tobase64, frombase64 options for encoding variants.
$ echo -n hello | base64 | fq -rRs 'frombase64 | tostring'
hello
tohex, fromhextoxmlentities, fromxmlentitiestourl, fromurl
$ echo -n 'https://host/path/?key=value#fragment' | fq -Rs 'fromurl | ., (.host = "changed" | tourl)'
{
"fragment": "fragment",
"host": "host",
"path": "/path/",
"query": {
"key": "value"
},
"rawquery": "key=value",
"scheme": "https"
}
"https://changed/path/?key=value#fragment"
tourlpath, fromurlpathtourlencode, fromurlencodetourlquery, fromurlquerytomd4tomd5
$ echo -n hello | fq -rRs 'tomd5 | tohex'
5d41402abc4b2a76b9719d911017c592
tosha1tosha256tosha512tosha3_224tosha3_256tosha3_384tosha3_512toiso8859_1, fromiso8859_1tutf8, fromutf8toutf16, fromutf16toutf16le, fromutf16letoutf16be, fromutf16be
$ echo -n 00680065006c006c006f | fq -rRs 'fromhex | fromutf16be'
hello
group function, same as group_by(.) #299<some object JQValue> as {$key})tojson($opts) that support indent option. tojson still works as before (no indent).
$ echo '{a: 1}' | fq -r 'tojson({indent: 2})'
{
"a": 1
}
--decode-file (will still work) to --argdecode be be more consistent with existing --arg* arguments. #309bitcoin Add Bitcoin blkdat, block, transcation and script decoders #239elf Use correct offset to dynamic linking string table #304tcp Restructure into separate client/server objects and add skipped_bytes (number of bytes with known missing ACK), has_start (has first byte in stream) and has_end (has last byte in stream) per direction #251
│00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection
│ │ │ source_ip: "192.168.69.2"
│ │ │ source_port: 34059
│ │ │ destination_ip: "192.168.69.1"
│ │ │ destination_port: "http" (80) (World Wide Web HTTP)
│ │ │ has_start: true
│ │ │ has_end: true
0x000│47 45 54 20 2f 74 65 73│GET /tes│ client_stream: raw bits
0x008│74 2f 65 74 68 65 72 65│t/ethere│
* │until 0x1bc.7 (end) (44│ │
0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│ server_stream: raw bits
0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│
* │until 0x191.7 (end) (40│ │
│00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection
│ │ │ client{}:
│ │ │ ip: "192.168.69.2"
│ │ │ port: 34059
│ │ │ has_start: true
│ │ │ has_end: true
│ │ │ skipped_bytes: 0
0x000│47 45 54 20 2f 74 65 73│GET /tes│ stream: raw bits
0x008│74 2f 65 74 68 65 72 65│t/ethere│
* │until 0x1bc.7 (end) (44│ │
│ │ │ server{}:
│ │ │ ip: "192.168.69.1"
│ │ │ port: "http" (80) (World Wide Web HTTP)
│ │ │ has_start: true
│ │ │ has_end: true
│ │ │ skipped_bytes: 0
0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│ stream: raw bits
0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│
* │until 0x191.7 (end) (40│ │
zip Add 64-bit support and add uncompress option #278matroska Update and regenerate based on latest spec and also handle unknown ids better #291mp4 Changes:
cat init frag | fq ....senc box support #290track_id to tracks #254trun data offset #294mvhd, tkhd, mdhd and mehd version 1 support #258fairplay_spc Add basic FairPlay Server Playback Context decoder #310avc_pps Correctly check for more rbsp datafq -d decode_samples=false d file.mp4 or ... | mp4({decode_samples: false}})fq -h mp4 or help(mp4) in the REPL.amf0 Add Action Message Format 0 decoder #214hevc_pps Add H.265/HEVC Picture Parameter Set decoder #210hevc_sps Add H.265/HEVC Sequence Parameter Set decoder #210hevc_vpc Add H.265/HEVC Video Parameter Set decoder #210mp3 Add max_unique_header_config and max_sync_seek options #242mp4 Simplify granule structure a bit #242mp4 Add decode_samples and allow_truncate options #242flac_frame Add bits_per_sample option #242icmpv6 Add Internet Control Message Protocol v6 decoder #216id3v2 Add v2.0 PIC supportipv6_packet Add Internet protocol v6 packet decoder #216macho Remove redundant arch struct level and cleanup some sym values #226macho Add raw fields for section and encryption info #238mp4 Add more HEIF boxes support #221mpeg_pes Support MPEG1 #219rtmp Add Real-Time Messaging Protocol decoder. Only plain RTMP for now. #214matroska Symbol name cleanup #220tcp Better port matching and make it possible to know if byte stream has start/end. #223udp Better port matching #223Added macho decoder (thanks @Akaame), nicer REPL interrupt, error and prompt, add slurp/spew functions and explode for binary.
Added fq talk slides from Binary Tools Summit 2022 to README.md.
repl, slurp and future help system. #178try ... | ... catch precedence issue.tonumber now supports non-base-10 numbers.slurp/spew to collect outputs and outputs them later.
1,2,3 | slurp("a") collects, later do spew("a") to output them. Also a global array $a will be available. #178explode for binary. #188
"a" | tobits | explode return bits [0,1,1,0,0,0,0,1]."åäö" | tobytes | explode return utf8 bytes instead of codepoints [195,165,195,164,195,182].--help: #177
--formats with --help formats. #181--help options to see all default option values. #181var, use slupr instead.macho Add decoder. Thanks @Akaame #43mp4 Support colr box. #176Improved binary slicing and bit reading, avro_ocf decoder (thanks @xentripetal), asn1_ber decoder, renamed display aliases, new grep_by and paste function.
bitio package.bitio to decode package.[0,1,1,0,0,1,1,0,0,1,1,1,0,0,0,1 | tobits] | tobytes | tostring returns "fq".[.frames[0], .frames[-1]] | mp3 decode mp3 based on first and last frame of other mp3.grep_by that recursively selects using a filter condition and ignores errors. #102
grep_by(.type == "trak") finds all objects where .type is "trak" (all mp4 track boxes).grep_by(tonumber | . >= 40 and . <= 100) find all numbers between 40 and 100.grep_by(format == "jpeg") find all jpegs.paste function to read string from stdin util ^D (EOF). Can be used to paste in REPl etc. #143
paste | frompem | asn1_ber | repl wait for PEM encoded text on stdin (paste and press ^D), decode it with asn1_ber and start a sub-REPL with the result.paste | fromjson decode pasted JSON.eval(paste) eval pasted jq expression.v and f, add da, dd, dv and ddv. #112
d/d($opts) display value and truncate long arrays and buffersda/da($opts) display value and don't truncate arraysdd/dd($opts) display value and don't truncate arrays or buffersdv/dv($opts) verbosely display value and don't truncate arrays but truncate buffersddv/ddv($opts) verbosely display value and don't truncate arrays or buffersradix into toradix($base)/fromradix($base). #139number_to_bytes. Can be done with tobytes. #139tobytes to zero pad most significant bits to byte alignment if needed. #133tobytes/tobits variant that takes an argument to add extra padding. #133
0xf | tobytes 8 bit binary with last 4 bits set0xf | tobytes(4) 32 bit binary with last 4 bits set0xf | tobits(12) 12 bit binary with last 4 bits settopem/frompem to work with PEM encoding. #92display, decode value, binary and binary array documentation. #118 #136 #133-o force=<bool> option that sets force decode option. Same as mp4({force: true}). #126avro_ocf Add decoder. #38 Thanks @xentripetal
asn1_ber Add decoder. #92
torepr support.aac_frame Only decode object types we know about. #130mp3 Shorter sync find heuristics. #104mp4 Add stz2 supportmp4 Add pnot (preview container) and jP (JPEG 2000) signature. #125Also thanks to @Doctor-love for keeping things tidy.
(Some commits have been removed from list for clarity)
torepr/0 function that converts decoded value into what it represents.
Initially works for:bencode, cbor, bson and msgpack #74
Example usage:
fq torepr file.cbor
fq -i torepr file.cbor
fq torepr.field file.cbor
fq 'torepr | .field | ...' file.cborstderr/0 function for jq compatibility #58band, bor etc are now normal functions instead of operators. Was done to be syntax compatible with jq #61
Uses the same convention as jq math functions: Unary uses input 123 | bnot, more than one argument all as arguments band(123; 456)bencode Add decoder #64cbor Add decoder #75msgpack Add decoder #69mp4 Much improved sample decode #82 #78png Decode PLTE and tRNS chunks #59tar Don't assume there is a end marker and support more than 2 blocks #86 #87Also thanks to @Doctor-love for keeping things tidy.
(Some commits have been removed from list for clarity)
print/0, println/0 function now properly convert input to string if needed. #54match functions now don't try to be smart with buffers, use. tobytes etc instead. Less confusing and also unbreak grep:ing decode values. #53
Now this works: fq 'grep("^strtab$") | parent | {name, string}' /bin/lsbson Add Binary JSON deccoderar Add Unix archive decoderbsd_loopback_frame Add BSD lookback frame decoder (used in pcap files)elf Now does a two-pass decode to correctly handle string table referenceself Decode more sections: symbol tables and hashesmatroska Assert sane tag size only for stringspcap Don't fail if incl_len > spanlenAlso thanks to @Doctor-love @mathieu-aubin for keeping things tidy.
(Some commits have been removed from list for clarity)