Task
coreutils-static-alpine
GNU coreutils 9.7 is a collection of Linux utilities like ls, cp, mv, etc.
Project link: gnu.org/software/coreutils
Task: Compile and install all coreutils utilities to a specific location. Compile them statically.
Difficulties: Static linking requires correctly configuring the build. Alpine Linux is less standard than Ubuntu.
Environment: Alpine Linux 3.22.1 on amd64, without internet access.
Best attempt
| Model |
qwen3-max
|
| Attempt |
p1dflhxaf1v9e
|
| # of commands | 3 |
| Total time | 58s |
| Cost | $0.009 |
Terminal transcript >
compile-bench> tar -xzf coreutils.tar.gz -C /home/peter [empty output] compile-bench> cd coreutils-9.7 && ./configure --prefix=/home/peter/result --enable-static --disable-shared LDFLAGS="-static" && make -j$(nproc) && make install make[5]: Entering directory '/home/peter/coreutils-9.7/gnulib-tests' make[5]: Leaving directory '/home/peter/coreutils-9.7/gnulib-tests' make[4]: Leaving directory '/home/peter/coreutils-9.7/gnulib-tests' make[3]: Leaving directory '/home/peter/coreutils-9.7/gnulib-tests' make[2]: Leaving directory '/home/peter/coreutils-9.7/gnulib-tests' make[1]: Leaving directory '/home/peter/coreutils-9.7' compile-bench> cd /home/peter/result/bin && for util in *; do ln -sf /home/peter/result/bin/$util /home/peter/result/$util; done [empty output]
Model ranking
| # | Model | ||||
|---|---|---|---|---|---|
| 1 |
qwen3-max
|
100%
|
5 (1.0x) | 59s (1.1x) | $0.021 (5.7x) |
| 2 |
gpt-4.1-mini
|
100%
|
6 (1.2x) | 56s (1.0x) | $0.005 (1.4x) |
| 3 |
gemini-2.5-pro
|
100%
|
7 (1.4x) | 2m3s (2.3x) | $0.044 (11.6x) |
| 4 |
grok-4-fast
|
100%
|
8 (1.6x) | 1m45s (1.9x) | $0.004 (1.0x) |
| 5 |
gpt-5-codex-high
|
100%
|
9 (1.8x) | 1m22s (1.5x) | $0.044 (11.7x) |
| 6 |
glm-4.5
|
100%
|
11 (2.2x) | 1m26s (1.6x) | $0.036 (9.4x) |
| 7 |
claude-opus-4.1-thinking-16k
|
100%
|
14 (2.8x) | 2m24s (2.7x) | $0.562 (149.3x) |
| 8 |
gemini-2.5-flash
|
100%
|
14 (2.8x) | 2m42s (3.0x) | $0.016 (4.2x) |
| 9 |
grok-code-fast-1
|
100%
|
14 (2.8x) | 4m26s (4.9x) | $0.013 (3.5x) |
| 10 |
claude-sonnet-4-thinking-16k
|
100%
|
18 (3.6x) | 2m26s (2.7x) | $0.164 (43.7x) |
| 11 |
claude-sonnet-4.5-thinking-16k
|
100%
|
21 (4.2x) | 2m37s (2.9x) | $0.168 (44.5x) |
| 12 |
claude-sonnet-4
|
100%
|
23 (4.6x) | 2m0s (2.2x) | $0.212 (56.3x) |
| 13 |
kimi-k2-0905
|
100%
|
25 (5.0x) | 2m12s (2.4x) | $0.257 (68.1x) |
| 14 |
claude-sonnet-4.5
|
100%
|
26 (5.2x) | 1m56s (2.1x) | $0.172 (45.8x) |
| 15 |
gpt-5-mini-minimal
|
67%
|
5 (1.0x) | 1m9s (1.3x) | $0.004 (1.1x) |
| 16 |
gemini-2.5-flash-thinking
|
67%
|
5 (1.0x) | 1m15s (1.4x) | $0.009 (2.3x) |
| 17 |
grok-4
|
67%
|
9 (1.8x) | 6m16s (6.9x) | $0.247 (65.6x) |
| 18 |
gpt-4.1
|
67%
|
10 (2.0x) | 54s (1.0x) | $0.037 (9.7x) |
| 19 |
gpt-5-mini-high
|
67%
|
10 (2.0x) | 2m52s (3.2x) | $0.018 (4.8x) |
| 20 |
deepseek-v3.1
|
67%
|
11 (2.2x) | 1m8s (1.3x) | $0.031 (8.3x) |
| 21 |
gpt-oss-120b-high
|
67%
|
11 (2.2x) | 1m10s (1.3x) | $0.006 (1.6x) |
| 22 |
gpt-5-high
|
67%
|
11 (2.2x) | 3m17s (3.6x) | $0.091 (24.1x) |
| 23 |
deepseek-v3.1-terminus
|
67%
|
19 (3.8x) | 2m2s (2.3x) | $0.049 (12.9x) |
| 24 |
gpt-5-minimal
|
33%
|
12 (2.4x) | 1m20s (1.5x) | $0.025 (6.5x) |
Attempts
| Model | Attempt ID | Cost | Duration | Status / Error |
|---|---|---|---|---|
|
claude-opus-4.1-thinking-16k
|
387hqtqn6lzy9
|
$0.419 | 2m23s | Success |
|
claude-opus-4.1-thinking-16k
|
bx2ctc0ns2ynx
|
$0.566 | 2m32s | Success |
|
claude-opus-4.1-thinking-16k
|
im394kaoa21dw
|
$0.562 | 2m24s | Success |
|
claude-sonnet-4
|
5ix2jj92xzvso
|
$0.169 | 2m0s | Success |
|
claude-sonnet-4
|
dwl0zf27xcptq
|
$0.219 | 1m53s | Success |
|
claude-sonnet-4
|
yac8cik20ldt9
|
$0.212 | 2m0s | Success |
|
claude-sonnet-4-thinking-16k
|
22jwht0d2ipal
|
$0.146 | 2m17s | Success |
|
claude-sonnet-4-thinking-16k
|
hc1lomwrt4zzu
|
$0.164 | 2m26s | Success |
|
claude-sonnet-4-thinking-16k
|
my5fotorj8aq1
|
$0.175 | 2m32s | Success |
|
claude-sonnet-4.5
|
3683h3teclfkp
|
$0.261 | 2m41s | Success |
|
claude-sonnet-4.5
|
5eoed85941nlj
|
$0.160 | 1m48s | Success |
|
claude-sonnet-4.5
|
wa5euy18grcn1
|
$0.172 | 1m56s | Success |
|
claude-sonnet-4.5-thinking-16k
|
7uxztvb87rn1i
|
$0.151 | 2m28s | Success |
|
claude-sonnet-4.5-thinking-16k
|
kxem1jnyg0fh5
|
$0.225 | 3m30s | Success |
|
claude-sonnet-4.5-thinking-16k
|
m004cc1c5rut0
|
$0.168 | 2m37s | Success |
|
deepseek-v3.1
|
77mv5k049r2hu
|
$0.049 | 1m22s | Success |
|
deepseek-v3.1
|
pxbr9fpz4dxt0
|
$0.031 | 1m8s | Success |
|
deepseek-v3.1
|
w2ojjoekpdbww
|
$0.030 | 1m8s | Failure task failed: sha1sum binary does not exist |
|
deepseek-v3.1-terminus
|
0ane20p9h27su
|
$0.148 | 5m1s | Success |
|
deepseek-v3.1-terminus
|
auu4ae4z7jng1
|
$0.049 | 2m2s | Success |
|
deepseek-v3.1-terminus
|
q3riyy2qid78w
|
$0.107 | 4m21s | Failure task failed: install missing at /home/peter/result/install or not executable |
|
gemini-2.5-flash
|
2l89o13kpizbx
|
$0.015 | 2m47s | Success |
|
gemini-2.5-flash
|
b1gb6tr8rufu1
|
$0.016 | 2m42s | Success |
|
gemini-2.5-flash
|
z4aj770xugw7i
|
$0.024 | 1m50s | Success |
|
gemini-2.5-flash-thinking
|
4lo9tks6n9tch
|
$0.009 | 1m15s | Success |
|
gemini-2.5-flash-thinking
|
oiv05619mgyhw
|
$0.016 | 2m51s | Success |
|
gemini-2.5-flash-thinking
|
qfi89k9nn1x5x
|
$0.120 | 14m58s | Failure LLM call failed: context deadline exceeded |
|
gemini-2.5-pro
|
8b3txg4az3fim
|
$0.044 | 2m1s | Success |
|
gemini-2.5-pro
|
jn27ogdi6pu8a
|
$0.056 | 2m3s | Success |
|
gemini-2.5-pro
|
mj9b3nav57hki
|
$0.042 | 2m4s | Success |
|
glm-4.5
|
ktr4qqs8xo0su
|
$0.063 | 1m37s | Success |
|
glm-4.5
|
p31pmdt10q55c
|
$0.034 | 1m26s | Success |
|
glm-4.5
|
ueue8k95q5iy3
|
$0.036 | 1m17s | Success |
|
gpt-4.1
|
7hmaishv71n24
|
$0.037 | 54s | Success |
|
gpt-4.1
|
u0vlvwt4jij3k
|
$0.028 | 57s | Failure task failed: kill missing at /home/peter/result/kill or not executable |
|
gpt-4.1
|
x2d4ka9mzebb0
|
$0.052 | 1m9s | Success |
|
gpt-4.1-mini
|
41v4f1oidpjf3
|
$0.005 | 55s | Success |
|
gpt-4.1-mini
|
fiieaupeprxhx
|
$0.005 | 57s | Success |
|
gpt-4.1-mini
|
w2nie94mxcrcu
|
$0.005 | 56s | Success |
|
gpt-5-codex-high
|
4oo5fq9i0d2y1
|
$0.069 | 2m0s | Success |
|
gpt-5-codex-high
|
5rir6cs6ipqj5
|
$0.039 | 1m19s | Success |
|
gpt-5-codex-high
|
mh3kfcatopd92
|
$0.044 | 1m22s | Success |
|
gpt-5-high
|
4dplv2asuz96k
|
$0.174 | 4m29s | Success |
|
gpt-5-high
|
6kcstptn9q8fu
|
$0.091 | 3m17s | Success |
|
gpt-5-high
|
nwmke9yqrdnee
|
$0.437 | 9m34s | Failure exceeded max tool calls (50) |
|
gpt-5-mini-high
|
1ympgcgxadamv
|
$0.016 | 2m57s | Failure task failed: sha1sum binary does not exist |
|
gpt-5-mini-high
|
72yxfisixfxth
|
$0.018 | 2m52s | Success |
|
gpt-5-mini-high
|
dc3kc0bwyz0bk
|
$0.021 | 3m17s | Success |
|
gpt-5-mini-minimal
|
gzm9ox4kjjy9g
|
$0.006 | 1m21s | Failure task failed: kill missing at /home/peter/result/kill or not executable |
|
gpt-5-mini-minimal
|
jxbbw55e1jaqj
|
$0.005 | 1m9s | Success |
|
gpt-5-mini-minimal
|
my8luqikggpov
|
$0.004 | 1m51s | Success |
|
gpt-5-minimal
|
0pwxxt2o80rcy
|
$0.030 | 1m19s | Failure task failed: kill missing at /home/peter/result/kill or not executable |
|
gpt-5-minimal
|
28d0p3p2ov9xj
|
$0.025 | 1m50s | Failure task failed: install missing at /home/peter/result/install or not executable |
|
gpt-5-minimal
|
q1kpb9n4d085m
|
$0.025 | 1m20s | Success |
|
gpt-oss-120b-high
|
fuij3ndjw20n5
|
$0.006 | 1m19s | Success |
|
gpt-oss-120b-high
|
fzaoi5zvmombl
|
$0.000 | 3s | Failure task failed: sha1sum binary does not exist |
|
gpt-oss-120b-high
|
zcvdwb7ywu2xx
|
$0.007 | 1m10s | Success |
|
grok-4
|
d73avs395xe5y
|
$0.247 | 6m16s | Success |
|
grok-4
|
s7aguiggk23s5
|
$0.752 | 9m2s | Success |
|
grok-4
|
yzdb8yjou3nn8
|
$0.254 | 14m57s | Failure context timeout: context deadline exceeded |
|
grok-4-fast
|
q7lrj6gon8oz3
|
$0.003 | 1m45s | Success |
|
grok-4-fast
|
tvib0otijpp5z
|
$0.004 | 1m30s | Success |
|
grok-4-fast
|
ty6wpc1id21sa
|
$0.004 | 1m50s | Success |
|
grok-code-fast-1
|
cwfpgy4vdsa4s
|
$0.017 | 4m51s | Success |
|
grok-code-fast-1
|
efk11pxkgc6v1
|
$0.013 | 4m26s | Success |
|
grok-code-fast-1
|
incylckzw6jd3
|
$0.005 | 1m7s | Success |
|
kimi-k2-0905
|
1j47xy70171ku
|
$0.257 | 2m12s | Success |
|
kimi-k2-0905
|
ad99wlmdq6o52
|
$0.134 | 1m36s | Success |
|
kimi-k2-0905
|
g3ais23ic9lc3
|
$0.368 | 2m52s | Success |
|
qwen3-max
|
d7cgr168yrcdq
|
$0.021 | 1m50s | Success |
|
qwen3-max
|
p1dflhxaf1v9e
|
$0.009 | 58s | Success |
|
qwen3-max
|
pse138dhuth27
|
$0.021 | 59s | Success |