Task
cowsay
Cowsay 3.8.4 is an ASCII-art speech bubble generator. Project link: github.com/cowsay-org/cowsay.
Task: Install the cowsay package to a specific location.
Difficulties: Since cowsay is just a single Perl script it doesn't require any compilation, however it comes with several asset files that need to be copied as well.
Environment: Ubuntu 22.04 on amd64, without internet access.
Best attempt
| Model |
gpt-5-mini-minimal
|
| Attempt |
d3uhbkoymgwyz
|
| # of commands | 4 |
| Total time | 18s |
| Cost | $0.002 |
Terminal transcript >
compile-bench> rm -rf /home/peter/result && mkdir -p /home/peter/result && cd /home/peter && tar -xzf cowsay.tar.gz && cd cowsay-3.8.4 || cd cowsay || exit 1 && ls -la && chmod +x configure || true && ./configure --prefix=/home/peter/result || true; make -j$(nproc) || make || true; make install DESTDIR=/home/peter/tmpinst || true; if [ -f /home/peter/tmpinst/home/peter/result/bin/cowsay ]; then mkdir -p /home/peter/result/bin && mv /home/peter/tmpinst/home/peter/result/bin/cowsay /home/peter/result/bin/; elif [ -f /home/peter/tmpinst/usr/local/bin/cowsay ]; then mkdir -p /home/peter/result/bin && mv /home/peter/tmpinst/usr/local/bin/cowsay /home/peter/result/bin/; fi; if [ ! -f /home/peter/result/bin/cowsay ]; then make install || true; fi; if [ -f /home/peter/result/bin/cowsay ]; then ln -sf /home/peter/result/bin/cowsay /home/peter/result/cowsay; fi; /bin/ls -la /home/peter/result || true install -c -d /home/peter/tmpinst/usr/local/share/cowsay/site-cows total 12 drwxr-xr-x 3 peter peter 4096 Sep 16 21:57 . drwxr-x--- 1 peter peter 4096 Sep 16 21:57 .. drwxr-xr-x 2 peter peter 4096 Sep 16 21:57 bin lrwxrwxrwx 1 peter peter 29 Sep 16 21:57 cowsay -> /home/peter/result/bin/cowsay compile-bench> /bin/ls -la /home/peter/result/bin && /home/peter/result/bin/cowsay "hello" | head -n3 total 20 drwxr-xr-x 2 peter peter 4096 Sep 16 21:57 . drwxr-xr-x 3 peter peter 4096 Sep 16 21:57 .. -rwxr-xr-x 1 peter peter 9713 Sep 16 21:57 cowsay cowsay: Could not find cowfile for 'default.cow'! compile-bench> mkdir -p /home/peter/result/share/cowsay/cows && cp -a share/cowsay/cows/*.cow /home/peter/result/share/cowsay/cows || true; /home/peter/result/bin/cowsay "hello" | head -n3 cp: cannot stat 'share/cowsay/cows/*.cow': No such file or directory cowsay: Could not find cowfile for 'default.cow'! compile-bench> cp -a /home/peter/cowsay-3.8.4/share/cowsay/cows /home/peter/result/share/cowsay || cp -a /home/peter/cowsay/share/cowsay/cows /home/peter/result/share/cowsay || true; ls -la /home/peter/result/share/cowsay/cows || true; /home/peter/result/bin/cowsay "hi" | head -n3 -rw-r--r-- 1 peter peter 213 Dec 1 2024 vader-koala.cow -rw-r--r-- 1 peter peter 279 Dec 1 2024 vader.cow -rw-r--r-- 1 peter peter 248 Dec 1 2024 www.cow ____ < hi > ----
Model ranking
| # | Model | ||||
|---|---|---|---|---|---|
| 1 |
gemini-2.5-pro
|
100%
|
6 (1.5x) | 27s (3.2x) | $0.024 (13.8x) |
| 2 |
grok-4-fast
|
100%
|
7 (1.8x) | 29s (3.4x) | $0.002 (1.0x) |
| 3 |
gpt-5-mini-high
|
100%
|
7 (1.8x) | 1m10s (8.2x) | $0.009 (5.2x) |
| 4 |
gpt-5-high
|
100%
|
7 (1.8x) | 1m16s (9.0x) | $0.045 (26.2x) |
| 5 |
gemini-2.5-flash-thinking
|
100%
|
8 (2.0x) | 17s (2.0x) | $0.005 (2.7x) |
| 6 |
grok-4
|
100%
|
8 (2.0x) | 1m26s (10.1x) | $0.112 (64.8x) |
| 7 |
gpt-4.1
|
100%
|
9 (2.2x) | 14s (1.6x) | $0.016 (9.2x) |
| 8 |
grok-code-fast-1
|
100%
|
9 (2.2x) | 23s (2.7x) | $0.004 (2.4x) |
| 9 |
gpt-5-codex-high
|
100%
|
9 (2.2x) | 38s (4.4x) | $0.030 (17.1x) |
| 10 |
claude-opus-4.1-thinking-16k
|
100%
|
10 (2.5x) | 1m28s (10.3x) | $0.367 (211.4x) |
| 11 |
glm-4.5
|
100%
|
11 (2.8x) | 53s (6.2x) | $0.016 (9.4x) |
| 12 |
claude-sonnet-4.5-thinking-16k
|
100%
|
12 (3.0x) | 50s (5.9x) | $0.070 (40.5x) |
| 13 |
claude-sonnet-4.5
|
100%
|
14 (3.5x) | 37s (4.4x) | $0.062 (35.7x) |
| 14 |
claude-sonnet-4-thinking-16k
|
100%
|
14 (3.5x) | 1m4s (7.5x) | $0.083 (47.6x) |
| 15 |
claude-sonnet-4
|
100%
|
15 (3.8x) | 44s (5.2x) | $0.055 (31.7x) |
| 16 |
deepseek-v3.1-terminus
|
100%
|
16 (4.0x) | 56s (6.6x) | $0.012 (6.9x) |
| 17 |
kimi-k2-0905
|
100%
|
19 (4.8x) | 38s (4.4x) | $0.056 (32.0x) |
| 18 |
gpt-5-minimal
|
67%
|
5 (1.2x) | 15s (1.8x) | $0.011 (6.1x) |
| 19 |
gpt-oss-120b-high
|
67%
|
7 (1.8x) | 9s (1.0x) | $0.002 (1.0x) |
| 20 |
qwen3-max
|
67%
|
7 (1.8x) | 13s (1.5x) | $0.011 (6.3x) |
| 21 |
deepseek-v3.1
|
67%
|
7 (1.8x) | 16s (1.8x) | $0.004 (2.2x) |
| 22 |
gemini-2.5-flash
|
67%
|
10 (2.5x) | 9s (1.1x) | $0.003 (2.0x) |
| 23 |
gpt-5-mini-minimal
|
33%
|
4 (1.0x) | 18s (2.1x) | $0.002 (1.4x) |
| 24 |
gpt-4.1-mini
|
33%
|
12 (3.0x) | 17s (2.0x) | $0.004 (2.3x) |
Attempts
| Model | Attempt ID | Cost | Duration | Status / Error |
|---|---|---|---|---|
|
claude-opus-4.1-thinking-16k
|
ab63zko4dieft
|
$0.316 | 1m26s | Success |
|
claude-opus-4.1-thinking-16k
|
l8ez2tqzf038e
|
$0.367 | 1m48s | Success |
|
claude-opus-4.1-thinking-16k
|
ngs999xy01yo0
|
$0.371 | 1m28s | Success |
|
claude-sonnet-4
|
0xcv93jcagtsg
|
$0.056 | 45s | Success |
|
claude-sonnet-4
|
15mx9azpgb1sf
|
$0.055 | 44s | Success |
|
claude-sonnet-4
|
2kn9xhss5mlu1
|
$0.055 | 44s | Success |
|
claude-sonnet-4-thinking-16k
|
0dk6j3q3ilrma
|
$0.080 | 1m4s | Success |
|
claude-sonnet-4-thinking-16k
|
bcfk66n0rhz6z
|
$0.092 | 1m17s | Success |
|
claude-sonnet-4-thinking-16k
|
pcm09yk5px3am
|
$0.083 | 1m4s | Success |
|
claude-sonnet-4.5
|
1e3wd93x8lqk6
|
$0.058 | 34s | Success |
|
claude-sonnet-4.5
|
cohtatz80lm3c
|
$0.062 | 37s | Success |
|
claude-sonnet-4.5
|
u9lgxwyv5ggbn
|
$0.073 | 38s | Success |
|
claude-sonnet-4.5-thinking-16k
|
0zuujyp1i8dw0
|
$0.090 | 1m2s | Success |
|
claude-sonnet-4.5-thinking-16k
|
ktl9xko2jsjsq
|
$0.069 | 50s | Success |
|
claude-sonnet-4.5-thinking-16k
|
y68c1g3pyuade
|
$0.070 | 50s | Success |
|
deepseek-v3.1
|
4qlmihewt9ufh
|
$0.076 | 1m24s | Failure task failed: Cowsay binary does not exist |
|
deepseek-v3.1
|
d4138dxenud78
|
$0.004 | 16s | Success |
|
deepseek-v3.1
|
v8z8a6awlcqge
|
$0.012 | 27s | Success |
|
deepseek-v3.1-terminus
|
8c908vm7vy1du
|
$0.023 | 1m28s | Success |
|
deepseek-v3.1-terminus
|
8ukqvxonjqb3o
|
$0.012 | 43s | Success |
|
deepseek-v3.1-terminus
|
jkldj1rwt0wam
|
$0.010 | 56s | Success |
|
gemini-2.5-flash
|
50cmjvmon2ep0
|
$0.005 | 11s | Failure task failed: Cowsay does not contain expected string (eyes) |
|
gemini-2.5-flash
|
row7obzz4gfgq
|
$0.007 | 17s | Success |
|
gemini-2.5-flash
|
v24mbpfq4luoe
|
$0.003 | 9s | Success |
|
gemini-2.5-flash-thinking
|
anorokmywf1rs
|
$0.006 | 21s | Success |
|
gemini-2.5-flash-thinking
|
mp2isanv6lc8r
|
$0.004 | 17s | Success |
|
gemini-2.5-flash-thinking
|
xbbddb0e2x7xm
|
$0.005 | 14s | Success |
|
gemini-2.5-pro
|
8q5s1d9mgqfly
|
$0.024 | 29s | Success |
|
gemini-2.5-pro
|
cmd5z7hrez88x
|
$0.018 | 25s | Success |
|
gemini-2.5-pro
|
uopqf502obo0v
|
$0.031 | 27s | Success |
|
glm-4.5
|
c7lwyzc7e35gi
|
$0.016 | 53s | Success |
|
glm-4.5
|
j5cjl9ny63ytv
|
$0.012 | 29s | Success |
|
glm-4.5
|
slu29ufqn4r3k
|
$0.017 | 59s | Success |
|
gpt-4.1
|
3id32xr9cey1p
|
$0.015 | 14s | Success |
|
gpt-4.1
|
abr84yraeyire
|
$0.016 | 13s | Success |
|
gpt-4.1
|
wy3uorgdtg3tq
|
$0.020 | 17s | Success |
|
gpt-4.1-mini
|
9awhy8x6j9ur4
|
$0.003 | 12s | Failure task failed: Cowsay does not contain expected string (eyes) |
|
gpt-4.1-mini
|
qsr9bokvjmfhz
|
$0.004 | 17s | Success |
|
gpt-4.1-mini
|
scf6zd1ml423x
|
$0.003 | 14m57s | Failure context timeout: context deadline exceeded |
|
gpt-5-codex-high
|
9tvhx8lbmjk6f
|
$0.020 | 35s | Success |
|
gpt-5-codex-high
|
hn2ban9rpomce
|
$0.036 | 42s | Success |
|
gpt-5-codex-high
|
rtyljyjzaev67
|
$0.030 | 38s | Success |
|
gpt-5-high
|
6o0f037m4ier7
|
$0.045 | 1m27s | Success |
|
gpt-5-high
|
8jb0vkfswvb49
|
$0.055 | 1m13s | Success |
|
gpt-5-high
|
t5td2t4tw673c
|
$0.037 | 1m16s | Success |
|
gpt-5-mini-high
|
5p07lpk4lcal9
|
$0.008 | 1m9s | Success |
|
gpt-5-mini-high
|
bogmtrlwbves7
|
$0.010 | 1m10s | Success |
|
gpt-5-mini-high
|
yuc2ey62bf6py
|
$0.009 | 1m20s | Success |
|
gpt-5-mini-minimal
|
d3uhbkoymgwyz
|
$0.002 | 18s | Success |
|
gpt-5-mini-minimal
|
er0nekqcuirte
|
$0.005 | 14m58s | Failure context timeout: context deadline exceeded |
|
gpt-5-mini-minimal
|
fpzyzsi4412rc
|
$0.003 | 14m58s | Failure context timeout: context deadline exceeded |
|
gpt-5-minimal
|
351q0x3pvxbvr
|
$0.011 | 15s | Success |
|
gpt-5-minimal
|
fyn7ooaajq14f
|
$0.011 | 15s | Success |
|
gpt-5-minimal
|
uicq8onf3sm5u
|
$0.023 | 35s | Failure task failed: Cowsay help does not contain expected string |
|
gpt-oss-120b-high
|
q6113qut3eep0
|
$0.000 | 2s | Failure task failed: Cowsay binary does not exist |
|
gpt-oss-120b-high
|
sxar3lb2frzjv
|
$0.002 | 9s | Success |
|
gpt-oss-120b-high
|
u640zwl6eozcm
|
$0.003 | 16s | Success |
|
grok-4
|
djw9wg2h6exct
|
$0.281 | 4m41s | Success |
|
grok-4
|
go3byi2cajhq9
|
$0.075 | 1m21s | Success |
|
grok-4
|
wqoodmrn3qfu8
|
$0.112 | 1m26s | Success |
|
grok-4-fast
|
5y1szq107gcl7
|
$0.002 | 27s | Success |
|
grok-4-fast
|
fav70mht93d78
|
$0.002 | 29s | Success |
|
grok-4-fast
|
si9j9koy5rxhd
|
$0.002 | 31s | Success |
|
grok-code-fast-1
|
410pjek3kwdj9
|
$0.004 | 23s | Success |
|
grok-code-fast-1
|
a1lci0c1z23x9
|
$0.006 | 41s | Success |
|
grok-code-fast-1
|
rciz7dso3pq9v
|
$0.003 | 19s | Success |
|
kimi-k2-0905
|
0rz3rz302eenu
|
$0.056 | 10m44s | Success |
|
kimi-k2-0905
|
3h3h3druudizt
|
$0.041 | 25s | Success |
|
kimi-k2-0905
|
8l8v39ygn9b57
|
$0.114 | 38s | Success |
|
qwen3-max
|
9sp1p5457r1nk
|
$0.027 | 21s | Success |
|
qwen3-max
|
ljjwc8rgd18z3
|
$0.008 | 12s | Failure task failed: Cowsay does not contain expected string (eyes) |
|
qwen3-max
|
yb4qz5ir33zts
|
$0.011 | 13s | Success |