Performance¶

Disk IO¶

Should files be read from multiple threads, even when the disk is the bottleneck? By having multiple concurrent reads, the operating system might be able to optimize the disk access pattern, and schedule reads more efficiently for higher throughput. Let’s measure.

This is for roughly 11500 files. Program output was redirected to /dev/null. Single-threaded taken from commit 4a5982ceb94b6a3dc575abce3c47e148dd28aa9f. Multi-threaded taken from commit cc06a48af7c8ea8b8b647443db1a26f77374f9e4.

Conclusion: multithreaded ingestion is advantageous, both when indexing from the disk, as well as when indexing from memory. There must be a CPU-bound part as well then. (On my system, for my workload, that is.) The next question then, is how much threads to use. 64 threads probably already takes all of the parallel gains, and the returns diminish quickly. It could be worth optimizing the number of threads for running time with a warm disk cache, and that would likely also perform almost optimally for a cold cache.

Some more results after reducing the thread queue size and doing non-blocking pushes, to keep the queue sizes more even:

And without queues or channels, protecting the directory iterator with a mutex instead:

Precollect¶

At commit c6c611be9179d939dc5646dc43ab8bdf5ddc2962, with 24 threads. First collecting discovered paths into a vec, and constructing the index by iterating over the paths in the vec. Is this the right thing to do, or should we put the paths iterator in a mutex directly? Measurement setup:

echo 3 | sudo tee /proc/sys/vm/drop_caches
perf stat target/release/musium ~/music

Note that the server was disabled to terminate the program after indexing. Also, these results are not comparable to the previous numbers, as the library has grown, and more data is processed. Furthermore, I did not redirect stdout to /dev/null in this case, but for a cold disk cache that does not make so much of a difference anyway.

Note that I did upgrade Walkdir when switching from vector precollect to the iterator-based version, so the comparison may be unfair. The data collected before the switch is labelled “Vec precollect 1”, the version after upgrading to Walkdir 2.1.4 is labelled “Vec precollect 2”. Furtherore, Walkdir 2.1.4 requires copying the path (labelled “double alloc”). I made a small change to the crate to be able to avoid the copy and extra allocation (labelled “single alloc”).

Counterintuitively, copying the path returned by the iterator is faster than not copying it. It might have something to do with ordering; spending more time in the iterator lock is actually a good thing? Or maybe I should collect more data, and this is just a statistical fluctuation. Just storing the paths is definitely faster if the copy is avoided:

copy    <- c(0.023223363, 0.022365082, 0.022318216, 0.022584837,
             0.020660742, 0.023839308, 0.022084252, 0.021812114,
             0.022180668, 0.019982074, 0.020979151, 0.023186709,
             0.024758619, 0.022889618, 0.024148854, 0.024708654)
noncopy <- c(0.022403112, 0.021863389, 0.019650964, 0.020984869,
             0.021901483, 0.021376926, 0.021668108, 0.021504715,
             0.023730031, 0.021861766, 0.021060567, 0.021986531,
             0.022680138, 0.019719019, 0.020053399, 0.021137137)
t.test(copy, noncopy)

#     Welch Two Sample t-test
#
# data:  copy and noncopy
# t = 2.6055, df = 28.297, p-value = 0.01447
# alternative hypothesis: true difference in means is not equal to 0
# 95 percent confidence interval:
#  0.000242829 0.002024684
# sample estimates:
#  mean of x  mean of y
# 0.02260764 0.02147388

So it is preferable to read many paths at once before processing them, perhaps due to better branch prediction. The gains are so big that the extra allocations and reallocations for storing the pathbuf pointers in a vec are totally worth it. It might be even better then to alternate beween scanning paths and processing them, to reduce peak memory usage, but let’s not worry about that at this point.

Fadvise¶

Command:

echo 3 | sudo tee /proc/sys/vm/drop_caches
perf stat target/release/musium cache /pool/music /pool/volatile/covers dummy

Measurements were performed with disks spinning. If the disks needed to spin up first, I restarted the measurement as soon as the disk was spinning.

Baseline, commit bcb01aac03b72c6250823d44d2b4dd71887e387c:

Open files first, read later, commit 0f2d00be7ef2009fe19af79ae02ac29d11c766cf:

Use “frontier” read pattern, commit 64371ff0aa834add77185531bae7160cfd6134ad:

After changing the IO queue size (and also tuning internal queue a bit), this could be brought down to 93 seconds, which suggests the win is really more in IO patterns, and for the warm case, simpler is probably better.

$ cat /sys/block/sd{b,c,d}/queue/nr_requests
4
4
4
$ echo 2048 | sudo tee /sys/block/sd{b,c,d}/queue/nr_requests
2048

Just regular blocking IO again, but tuning the disk queue size, commit a1ac4d2100f6d0bd61f0be99fc31b7da260be6af: