Ever wanted more different ways to understand what’s going on in a program? Here I catalogue a huge variety of tracing methods you can use for varying types of problems. Tracing has been such a long-standing interest (and job) of mine that some of these will novel and interesting to anyone who reads this. I’ll guarantee it by including 2 novel tracing tools I’ve made and haven’t shared before (loo

(This is based on Table 4.1 "Linux Crisis Tools" in SysPerf 2.) Some longer notes: [1] bcc and bpftrace have many overlapping tools: the bcc ones are more capable (e.g., CLI options), and the bpftrace ones can be edited on the fly. But that's not to say that one is better or faster than the other: They emit the same BPF bytecode and are equally fast once running. Also note that bcc is evolving and

The following content is rendered from my #til github.Linux Network PerformanceSource: https://github.com/leandromoreira/linux-network-performance-parameters/https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdfhttps://www.coverfire.com/articles/queueing-in-the-linux-network-stack/https://blog.cloudflare.com/how-to-achieve-low-latency/https://blog.cloudf

Last week during a casual conversation I overheard a colleague saying: "The Linux network stack is slow! You can't expect it to do more than 50 thousand packets per second per core!" That got me thinking. While I agree that 50kpps per core is probably the limit for any practical application, what is the Linux networking stack capable of? Let's rephrase that to make it more fun: On Linux, how hard

Fosdem, February 2021 Network Performance in the Linux Kernel Maxime Chevallier maxime.chevallier@bootlin.com © Copyright 2004-2021, Bootlin. Creative Commons BY-SA 3.0 license. Corrections, suggestions, contributions and translations are welcome! embedded Linux and kernel engineering - Kernel, drivers and embedded Linux - Development, consulting, training and support - https://bootlin.com 1/1 Max

The aim of this post is to point out potential kernel tunables that might improve network performance in certain scenarios. As with any other post on the subject, make sure you test before and after you make an adjustment to have a measurable, quantitative result. For the most part, the kernel is smart enough to detect and adjust certain TCP options after boot, or even dynamically, e.g the Sliding

Some years ago I have tested network interrupts affinity - you set ~0 as a CPU mask to balance network interrupts among all your CPU cores and you get all softirq instances running in parallel. Such interrupts distribution among CPU cores sometimes is a bad idea due to CPU caches computational burden and probable packets reordering. In most cases it is not recommended for servers performing some T