Task
jq

jq 1.8.1 is a command-line JSON utility for viewing and transforming JSON. Project link: github.com/jqlang/jq

Task: Compile and install jq to a specific location.

Difficulties: Standard autotools setup, nothing special.

Environment: Ubuntu 22.04 on amd64, without internet access.

Best attempt
Model gpt-5-high logo gpt-5-high
Attempt ugaa1u07fm2yh
# of commands 2
Total time 1m50s
Cost $0.066
Terminal transcript >
compile-bench> set -e; test -f jq.tar.gz; topdir=$(tar -tzf jq.tar.gz | head -n1 | cut -d/ -f1); rm -rf "$topdir"; tar -xzf jq.tar.gz; cd "$topdir"; if [ -f configure ]; then chmod +x configure || true; ./configure --prefix="/home/peter/result"; make -j"$(nproc)"; make install; elif [ -f CMakeLists.txt ]; then rm -rf build; cmake -S . -B build -DCMAKE_INSTALL_PREFIX="/home/peter/result" -DCMAKE_BUILD_TYPE=Release; cmake --build build -j"$(nproc)"; cmake --install build; else echo "No configure or CMakeLists.txt found"; exit 1; fi; mkdir -p "/home/peter/result"; if [ -x "/home/peter/result/bin/jq" ]; then ln -sf "/home/peter/result/bin/jq" "/home/peter/result/jq"; elif [ -x "/home/peter/result/usr/bin/jq" ]; then ln -sf "/home/peter/result/usr/bin/jq" "/home/peter/result/jq"; else echo "jq binary not found in install prefix"; exit 1; fi; "/home/peter/result/bin/jq" --version || "/home/peter/result/usr/bin/jq" --version; "/home/peter/result/jq" --version
 /usr/bin/install -c -m 644 libjq.pc '/home/peter/result/lib/pkgconfig'
make[3]: Leaving directory '/home/peter/jq-1.8.1'
make[2]: Leaving directory '/home/peter/jq-1.8.1'
make[1]: Leaving directory '/home/peter/jq-1.8.1'
jq-1.8.1
jq-1.8.1
compile-bench> ls -l /home/peter/result /home/peter/result/bin/jq /home/peter/result/jq; test -L /home/peter/result/jq && echo SYMLINK_OK || echo SYMLINK_MISSING; readlink -f /home/peter/result/jq; /home/peter/result/jq --version
lrwxrwxrwx 1 peter peter   25 Sep 16 21:29 jq -> /home/peter/result/bin/jq
drwxr-xr-x 3 peter peter 4096 Sep 16 21:29 lib
drwxr-xr-x 4 peter peter 4096 Sep 16 21:29 share
SYMLINK_OK
/home/peter/result/bin/jq
jq-1.8.1
Model ranking
# Model
1
gpt-5-mini-minimal logo gpt-5-mini-minimal
100%
3 (1.0x) 24s (1.0x) $0.002 (1.0x)
2
qwen3-max logo qwen3-max
100%
5 (1.7x) 26s (1.1x) $0.025 (14.7x)
3
gpt-5-minimal logo gpt-5-minimal
100%
5 (1.7x) 31s (1.4x) $0.013 (7.6x)
4
gpt-4.1-mini logo gpt-4.1-mini
100%
6 (2.0x) 37s (1.6x) $0.002 (1.4x)
5
gemini-2.5-flash-thinking logo gemini-2.5-flash-thinking
100%
7 (2.3x) 44s (2.0x) $0.007 (4.4x)
6
gemini-2.5-pro logo gemini-2.5-pro
100%
7 (2.3x) 56s (2.5x) $0.059 (35.1x)
7
gpt-5-high logo gpt-5-high
100%
7 (2.3x) 1m41s (4.5x) $0.051 (30.4x)
8
gpt-5-mini-high logo gpt-5-mini-high
100%
7 (2.3x) 1m48s (4.8x) $0.011 (6.5x)
9
grok-code-fast-1 logo grok-code-fast-1
100%
8 (2.7x) 43s (1.9x) $0.004 (2.2x)
10
grok-4 logo grok-4
100%
8 (2.7x) 1m50s (4.8x) $0.110 (65.4x)
11
gpt-oss-120b-high logo gpt-oss-120b-high
100%
9 (3.0x) 23s (1.0x) $0.004 (2.2x)
12
gpt-4.1 logo gpt-4.1
100%
9 (3.0x) 25s (1.1x) $0.030 (18.1x)
13
deepseek-v3.1 logo deepseek-v3.1
100%
9 (3.0x) 33s (1.4x) $0.026 (15.3x)
14
grok-4-fast logo grok-4-fast
100%
9 (3.0x) 1m1s (2.7x) $0.004 (2.6x)
15
claude-opus-4.1-thinking-16k logo claude-opus-4.1-thinking-16k
100%
9 (3.0x) 1m40s (4.4x) $0.437 (259.4x)
16
gpt-5-codex-high logo gpt-5-codex-high
100%
11 (3.7x) 47s (2.1x) $0.044 (26.2x)
17
claude-sonnet-4.5-thinking-16k logo claude-sonnet-4.5-thinking-16k
100%
12 (4.0x) 1m9s (3.0x) $0.104 (61.9x)
18
claude-sonnet-4-thinking-16k logo claude-sonnet-4-thinking-16k
100%
13 (4.3x) 1m16s (3.3x) $0.105 (62.1x)
19
deepseek-v3.1-terminus logo deepseek-v3.1-terminus
100%
14 (4.7x) 1m6s (2.9x) $0.024 (14.4x)
20
claude-sonnet-4.5 logo claude-sonnet-4.5
100%
15 (5.0x) 56s (2.5x) $0.105 (62.5x)
21
claude-sonnet-4 logo claude-sonnet-4
100%
16 (5.3x) 1m3s (2.8x) $0.141 (84.0x)
22
glm-4.5 logo glm-4.5
67%
11 (3.7x) 49s (2.2x) $0.027 (16.2x)
23
kimi-k2-0905 logo kimi-k2-0905
67%
15 (5.0x) 57s (2.5x) $0.112 (66.5x)
24
gemini-2.5-flash logo gemini-2.5-flash
33%
8 (2.7x) 35s (1.6x) $0.007 (4.1x)
Attempts
Model Attempt ID Cost Duration Status / Error
claude-opus-4.1-thinking-16k logo claude-opus-4.1-thinking-16k
2eseq5y7nrnff $0.437 1m40s Success
claude-opus-4.1-thinking-16k logo claude-opus-4.1-thinking-16k
mf2jn3tdak3lk $0.402 1m24s Success
claude-opus-4.1-thinking-16k logo claude-opus-4.1-thinking-16k
vebyzhzybqhi7 $0.591 2m0s Success
claude-sonnet-4 logo claude-sonnet-4
11nkif8m31k3w $0.141 1m3s Success
claude-sonnet-4 logo claude-sonnet-4
1smv7lunemq9c $0.133 57s Success
claude-sonnet-4 logo claude-sonnet-4
p16ou4n3u7rbv $0.167 1m28s Success
claude-sonnet-4-thinking-16k logo claude-sonnet-4-thinking-16k
0by7mbfywtdqx $0.121 1m23s Success
claude-sonnet-4-thinking-16k logo claude-sonnet-4-thinking-16k
21bccfjeuzxi5 $0.105 1m16s Success
claude-sonnet-4-thinking-16k logo claude-sonnet-4-thinking-16k
24iva3wt52cyr $0.103 1m9s Success
claude-sonnet-4.5 logo claude-sonnet-4.5
5nx8nmvqfr87g $0.105 56s Success
claude-sonnet-4.5 logo claude-sonnet-4.5
7q1klrbkzplre $0.096 52s Success
claude-sonnet-4.5 logo claude-sonnet-4.5
7xlfmjchjlief $0.141 1m8s Success
claude-sonnet-4.5-thinking-16k logo claude-sonnet-4.5-thinking-16k
7vhu3u6i9qkyi $0.090 59s Success
claude-sonnet-4.5-thinking-16k logo claude-sonnet-4.5-thinking-16k
97zr8snjnu5zo $0.139 1m10s Success
claude-sonnet-4.5-thinking-16k logo claude-sonnet-4.5-thinking-16k
twbyw2d8mhz0u $0.104 1m9s Success
deepseek-v3.1 logo deepseek-v3.1
2nokdzicmmfeu $0.026 31s Success
deepseek-v3.1 logo deepseek-v3.1
7qksvs0okfcth $0.026 35s Success
deepseek-v3.1 logo deepseek-v3.1
i6uvv4utdt04o $0.023 33s Success
deepseek-v3.1-terminus logo deepseek-v3.1-terminus
5w52z3uiohofo $0.017 1m6s Success
deepseek-v3.1-terminus logo deepseek-v3.1-terminus
ik4ih67xurqxc $0.024 1m2s Success
deepseek-v3.1-terminus logo deepseek-v3.1-terminus
yhzh0n7kcek3y $0.055 1m41s Success
gemini-2.5-flash logo gemini-2.5-flash
cp0eay12p46t7 $0.007 35s Success
gemini-2.5-flash logo gemini-2.5-flash
ipmliug5yiifz $0.000 1s Failure task failed: jq binary does not exist
gemini-2.5-flash logo gemini-2.5-flash
q7614ipz7drgv $0.009 28s Failure task failed: jq binary does not exist
gemini-2.5-flash-thinking logo gemini-2.5-flash-thinking
338s4eah51lk2 $0.007 44s Success
gemini-2.5-flash-thinking logo gemini-2.5-flash-thinking
74t14ujo90avy $0.006 42s Success
gemini-2.5-flash-thinking logo gemini-2.5-flash-thinking
e6g5iw6pk03kh $0.010 48s Success
gemini-2.5-pro logo gemini-2.5-pro
8knzqzdpnatex $0.059 56s Success
gemini-2.5-pro logo gemini-2.5-pro
cy85xpo9lr05z $0.028 48s Success
gemini-2.5-pro logo gemini-2.5-pro
ktb4un614oe96 $0.060 1m3s Success
glm-4.5 logo glm-4.5
4hazz0u6ks713 $0.027 58s Success
glm-4.5 logo glm-4.5
bcdzeowlwff5h $0.035 49s Success
glm-4.5 logo glm-4.5
eiq4lq8wiyu1l $0.018 44s Failure task failed: jq binary does not exist
gpt-4.1 logo gpt-4.1
qt6cp6j4m8oyp $0.028 23s Success
gpt-4.1 logo gpt-4.1
vjoaixidissnv $0.030 28s Success
gpt-4.1 logo gpt-4.1
x7pjvpldnf7b1 $0.033 25s Success
gpt-4.1-mini logo gpt-4.1-mini
81t1qjczdy22d $0.002 37s Success
gpt-4.1-mini logo gpt-4.1-mini
gnbm39zwmwk5n $0.002 37s Success
gpt-4.1-mini logo gpt-4.1-mini
r11pfzhpwmo4t $0.008 29s Success
gpt-5-codex-high logo gpt-5-codex-high
0kbtr8nfthbyp $0.044 47s Success
gpt-5-codex-high logo gpt-5-codex-high
158bgragokoer $0.045 51s Success
gpt-5-codex-high logo gpt-5-codex-high
83r05vg81hpci $0.023 41s Success
gpt-5-high logo gpt-5-high
sge8f0udia7a7 $0.048 1m32s Success
gpt-5-high logo gpt-5-high
ugaa1u07fm2yh $0.066 1m50s Success
gpt-5-high logo gpt-5-high
yzssxil81hmtk $0.051 1m41s Success
gpt-5-mini-high logo gpt-5-mini-high
9w2r3hjb3770q $0.010 2m5s Success
gpt-5-mini-high logo gpt-5-mini-high
aqtaxaa2l9l0q $0.011 1m48s Success
gpt-5-mini-high logo gpt-5-mini-high
shh2snps1fcz9 $0.013 1m29s Success
gpt-5-mini-minimal logo gpt-5-mini-minimal
26v39d6u13isn $0.002 23s Success
gpt-5-mini-minimal logo gpt-5-mini-minimal
fackcax423deu $0.004 24s Success
gpt-5-mini-minimal logo gpt-5-mini-minimal
g6w5myagubmml $0.002 24s Success
gpt-5-minimal logo gpt-5-minimal
gtzqzmmw2k1i9 $0.016 34s Success
gpt-5-minimal logo gpt-5-minimal
j0pc54flzix2u $0.010 31s Success
gpt-5-minimal logo gpt-5-minimal
q6v2sqchrus3x $0.013 24s Success
gpt-oss-120b-high logo gpt-oss-120b-high
fyfz6c4uffoei $0.004 24s Success
gpt-oss-120b-high logo gpt-oss-120b-high
gjt0kodeq85ph $0.001 17s Success
gpt-oss-120b-high logo gpt-oss-120b-high
rc5fl2hdq8gxw $0.005 23s Success
grok-4 logo grok-4
1fbpcmjw7hshm $0.120 1m50s Success
grok-4 logo grok-4
2et9jg34j40ag $0.092 1m36s Success
grok-4 logo grok-4
wufdaiz3qjkbo $0.110 2m3s Success
grok-4-fast logo grok-4-fast
48g4lp74wqo84 $0.004 58s Success
grok-4-fast logo grok-4-fast
lbofu9z26p8q3 $0.005 1m2s Success
grok-4-fast logo grok-4-fast
u059oqv5bunny $0.004 1m1s Success
grok-code-fast-1 logo grok-code-fast-1
bkqs3u14pl910 $0.003 41s Success
grok-code-fast-1 logo grok-code-fast-1
fgaujw38m8noi $0.004 43s Success
grok-code-fast-1 logo grok-code-fast-1
mqy53pxtx868v $0.004 48s Success
kimi-k2-0905 logo kimi-k2-0905
6l0e8jum54ze0 $0.005 12s Failure task failed: jq binary does not exist
kimi-k2-0905 logo kimi-k2-0905
9en79xikpk4ne $0.112 57s Success
kimi-k2-0905 logo kimi-k2-0905
nas9g5yxv5i0n $0.165 1m1s Success
qwen3-max logo qwen3-max
imvp0mb6m6ebc $0.025 26s Success
qwen3-max logo qwen3-max
sooogihyznrcc $0.025 39s Success
qwen3-max logo qwen3-max
x9zozgysnhqok $0.025 24s Success