Oversized executables

[alexchandel]

The uutils executables are a bit larger than their native counterparts. These are the stats on OS X with O3, LTO, and alloc_system:

Name	Native	uutils
base64	8.0K	200K
basename	8.0K	160K
cat	8.0K	204K
chmod	12K	524K
chroot	8.0K	220K
cksum	8.0K	180K
comm	8.0K	168K
cp	12K	204K
cut	8.0K	296K
dirname	8.0K	152K
du	12K	328K
echo	8.0K	132K
env	8.0K	148K
expand	8.0K	208K
expr	8.0K	168K
factor	-	208K
false	4.0K	80K
fold	8.0K	196K
groups	8.0K	168K
hashsum	-	596K
head	8.0K	196K
hostid	-	148K
hostname	8.0K	192K
id	8.0K	192K
kill	8.0K	180K
link	8.0K	156K
ln	8.0K	208K
logname	8.0K	156K
mkdir	8.0K	192K
mkfifo	8.0K	164K
mv	8.0K	220K
nice	8.0K	176K
nl	8.0K	512K
nohup	8.0K	184K
nproc	-	160K
od	16K	140K
paste	8.0K	200K
printenv	8.0K	160K
ptx	-	668K
pwd	8.0K	164K
readlink	12K	192K
realpath	-	192K
relpath	-	200K
rm	8.0K	208K
rmdir	8.0K	176K
seq	8.0K	228K
shuf	-	224K
sleep	8.0K	200K
sort	28K	220K
split	8.0K	228K
stdbuf	-	244K
sum	8.0K	180K
sync	8.0K	148K
tac	-	192K
tail	12K	200K
tee	8.0K	204K
test	8.0K	112K
timeout	-	264K
touch	8.0K	200K
tr	12K	196K
true	4.0K	80K
truncate	-	196K
tsort	8.0K	216K
tty	8.0K	160K
uname	8.0K	164K
unexpand	8.0K	212K
uniq	8.0K	220K
unlink	8.0K	164K
uptime	12K	204K
users	8.0K	160K
wc	8.0K	192K
whoami	8.0K	160K
yes	4.0K	160K

I think the funniest one is nl, which is 6300% larger than the native nl. jemalloc would've added another 230K to each of these.

I realize some of this is Rust's fault: when an optimized, LTO'd, alloc_system'd fn main(){println!("Hi!\n");} is still 84K, there's not much room. For example from the object dump/disassembly, about 9% of that dead weight was panicking code & string literals for the standard library :\ If we're really condemned to that, and to an 80K hello world, with all the implied overhead (and it's clearly to scale, as seen above), then this raises serious doubts about Rust as a system language.

But surely we can shed some of the remaining 196K/216K/etc off of tr/tsort/friends? The median size of the native executables is 8.0K.

[ebfe]

That's one of the reasons the multicall binary exists. As for individual binaries, I'm not sure what we can do except trying to reduce the number of dependencies.

[hexsel]

The project could look into https://github.com/lrs-lang/lib, but it looks like a pretty big change and it may remove large amounts of cross-platform support.

[ebfe]

@Hexel I did :) Unfortunately it's linux only.

[nathanross]

The elephants in the room are static linking, ABI compatibility, and as you mentioned libstd.

Some demonstrations. all "results" blocks are generated using the following command:

strip main && stat --printf="%s bytes\n" main && ldd main | cut -d= -f1

All of the results are going to be pretty specific to x86_64. I'm running debian stable.

All of the results below link against libc, the C standard library. You discuss suitability as a systems language. C is a system language, but most of its userland applications, like rust, use a standard library which takes up more than hundreds of kilobytes.

The difference as is demonstrated below, is that cargo by default does not use dynamic linking, because it's expected that most users will not (at this point in time) have an ABI-compatible rust standard library installed.

approaches

Rust, using libstd, static linking to libstd

command

echo 'fn main(){println!("Hello!\n");}' > main.rs
rustc -C opt-level=3 -C lto main.rs

results

290864 bytes
        linux-vdso.so.1 (0x00007fff6fd88000)
        libpthread.so.0 
        libgcc_s.so.1 
        libc.so.6 
        /lib64/ld-linux-x86-64.so.2 (0x00007f278dfd8000)

Rust, using libstd, dynamic linking to libstd

libstd is about 4.4mb and has to live on the operating system.

command

echo 'fn main(){println!("Hello!\n");}' > main.rs
rustc -C opt-level=3 -C prefer-dynamic main.rs

results

5528 bytes
        linux-vdso.so.1 (0x00007ffc9c70b000)
        libstd-17a8ccbd.so 
        libc.so.6 
        libdl.so.2 
        libpthread.so.0 
        libgcc_s.so.1 
        /lib64/ld-linux-x86-64.so.2 (0x00007f7e2c10c000)
        libm.so.6 
        librt.so.1 

Rust, no libstd framework

commands

wget https://raw.githubusercontent.com/rust-lang/rust/master/src/test/run-pass/smallest-hello-world.rs
rustc -C opt-level=3 -C lto -o main smallest-hello-world.rs

results

4992 bytes
        linux-vdso.so.1 (0x00007ffeae1bf000)
        libc.so.6 
        /lib64/ld-linux-x86-64.so.2 (0x00007f63f0832000)

C++, dynamic linking to libstdc++

Provided on my system by libgcc, which links directly against libm, so together they take up about 1.1mb total.

commands

echo -e '#include <iostream>\nint main() { std::cout << "Hello!\\n"; return 0; }' > main.cpp
clang++ -Oz main.cpp -o main

results

5400 bytes
        linux-vdso.so.1 (0x00007ffd4d58e000)
        libstdc++.so.6 
        libm.so.6 
        libgcc_s.so.1 
        libc.so.6 
        /lib64/ld-linux-x86-64.so.2 (0x00007f943e40b000)

C

commands

echo -e '#include <stdio.h>\nint main() { printf("Hello!\n"); }' > main.c
clang -Oz main.c -o main

results

4616 bytes
        linux-vdso.so.1 (0x00007fff4363f000)
        libc.so.6 
        /lib64/ld-linux-x86-64.so.2 (0x00007fc53febd000)

Can we just do everything dynamically

Imaginably if a distribution like debian or fedora has a libstdc++ that c++ programs can be compiled against, why not do this for rust?

Rust currently lacks ABI stability.

Well, it's the same and it isn't. C++ these uses a (relatively much more) stable ABI - it usually only changes with a major standards change. This means that when libstdc++ is compiled with a slightly newer version of clang you can install that on your system without also upgrading binaries that use libstdc++ and were compiled with an older version of clang.

When it has ABI stability, we'll be able to compete.

Rust doesn't have that yet, it's in the works as the Rust developers know it's needed to be able to ship binaries that are not so tightly coupled. But in the meantime, if you have a library that was complied with rustc v1.2 and you upgrade it and this new version is compiled with rustc v1.5, all binaries and libraries you were using that were linking against that library also now need to be replaced with versions of themselves compiled with rustc v1.5.

At some point in the future, there will be a stable ABI, and some systems will begin installing libstd as a dependency for some other tool. And for systems which have libstd, not only will the footprint of uutils coreutils per se is going to be a few hundred kb less, but we'll be able to painlessly split it into one binary per tool.

In the meantime

In the meantime, the best option is just as we're going right now - statically include parts of libstd that we need, then use a "multicall install" which imitates a binary for each tool we're providing via symbolic links.

In terms of speed, dynamic or static linking really has a negligible difference.

[nathanross]

creating linkback for #140

[alexchandel]

@nathanross Dynamic linking is not the answer, nor is multicall.

Rust programs already dynamically link to libSystem on OS X, which provides the entire C standard library plus a multitude of other features. The solution is not to dynamically link libstd.

The "features" libstd provides on top of libSystem are minimal—primarily structural—and in trivial programs ought to be removable. And indeed they can be, as LRS-lang demonstrates, but this requires undoing design flaws from Rust.

And multicall works rather poorly on windows. The solution instead is to judiciously code the binaries as close to the 80K minimum as possible.

[vadimcn]

@alexchandel: how would a C library help with providing e.g. Rust-style string formatting?

A viable solution on Windows might be multicall built as a dylib, plus a small stub binary that just calls the main entry point in that library. The latter could use #![no_std] to ensure the smallest possible size.

[alexchandel]

@vadimcn It doesn't need to, because string-formatting makes up relatively little of these binaries, and is inlined little with relative ease (once you stop panicking).

If you actually read the disassembly for PROFILE=release cp, you'll find that the largest symbols by far (at 34% of the text section) are __ZN4copy20h24e32c79ba610ccdJmaE and __ZN6uumain20ha15e8d6f5b9ddecfceaE. And you'll notice that there are calls to a huge number of symbols that panic in unoptimizable ways. Many of these are from show_error, which makes two unoptimizable writes that inexplicably panic instead of silently failing or aborting, but there are just as many explicit panics.

For comparison, I'm working on an ls for coreutils that never panics and obviously uses its own print/show_error macros, and it's barely 130K yet does far more than cp. I haven't even gotten around to optimizing it yet.

[nathanross]

fascinating @alexchandel your continued investigation into, and passion about, this topic is greatly appreciated.

[bachp]

@alexchandel Have you done any additional test since this issue was last discussed? It has been more than 2 years now and rust has changed quite a lot. Would be interesting to see how the binary size was influenced by this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.