The tools is best known in the bufferbloat community for it's test Realtime Response Under Load (RRUL), but the netperf-wrapper tool has other tests that I find useful.
Core software dependencies are recent versions of netperf, python, matplotlib and fping (optional are d-itg and http_runner).
Dependency issues on RHEL6First dependencies are solved easily by installing "python-matplotlib":
$ sudo yum install -y python-matplotlib python-pip
The software dependencies turned out to be a challenge on my RHEL6 box.
The "ping" program is too old to support option "-D" (prints timestamp before each-line). Work-around is to install "fping", which I choose to do from "rpmforge":
Commands needed for install "fping":
# rpm -Uvh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
# yum install -y fping
The "netperf" tool itself (on RHEL6) were not compiled with configure option "--enable-demo=yes" which is needed to get timestamp and continuous result output during a test-run.
Thus, I needed to recompile "netperf" manually:
- Download: ftp://ftp.netperf.org/netperf/netperf-2.6.0.tar.gz
- Extract and configure using: ./configure --enable-demo=yes --enable-histogram
- (Note, I also choose to enable histogram recording, which is not a requirement)
Install netperf-wrapperInstallation is quite simple, once the dependencies have been meet:
- git clone https://github.com/tohojo/netperf-wrapper.git
- cd netperf-wrapper
- sudo python2 setup.py install
GUI modeThere is a nice GUI mode for investigating and comparing results, started by:
$ netperf-wrapper --gui
This depend on matplotlib with Qt4 (and PyQt4), which unfortunately were not available for RHEL6. Fortunately there were a software package for this on Fedora, named "python-matplotlib-qt4".
For GUI mode netperf-wrapper needs: matplotlib with Qt4
$ sudo yum install -y python-matplotlib-qt4 PyQt4
Thus, the workflow is to run the tests on my RHEL6 machines, and analyze the result files on my Fedora laptop.
Using the toolThe same tool "netperf-wrapper" is both used for running the test, and later analyzing the result.
Listing the tests available:
$ netperf-wrapper --list-tests
For listing which plots are available for a given test e.g. "rrul":
$ netperf-wrapper --list-plots rrul
Before running a test towards a target system, remember to start the "netserver" daemon process on the target host (just run command "netserver" nothing else).
Start a test-run towards e.g. IP 192.168.8.2 with test rrul
$ netperf-wrapper -H 192.168.8.2 -t my_title rrul
It is recommend using the option "-t" to give your test a title, which makes is easier to distinguish when comparing two or more test files in e.g. the GUI tool.
The results of the test-run will be stored in a compressed json formatted text file, with the naming convention: rrul-2014-MM-DDTHHMMSS.milisec.json.gz
To view the result, without the GUI, run:
$ netperf-wrapper -i rrul_prio-2014-09-10T125650.993908.json.gz -f plot
Or e.g. selecting a specific plot like "ping_cdf"
$ netperf-wrapper -i rrul_prio-2014-09-10T125650.993908.json.gz -f plot -p ping_cdf
netperf-wrapper can also output numeric data suitable for plotting in org-mode or .csv (spreadsheet) format, but I didn't play with those options.
Updates: A release 0.7.0 of netperf-wrapper is pending
Extra: On bufferbloat
Interested in more about bufferbloat?
Too few people are linking to the best talk explaining bufferbloat and how it's solved by Van Jacobson (slides). The video quality is unfortunately not very good.
I've used some of Van's point in my own talk about bufferbloat: Beyond the existences of Bufferbloat, have we found the cure? (slides)
One problem with using this on the nanosec scale is the accuracy of your measurements, which depend on the accuracy of the hardware your are using.
Modern systems have power-saving and turbo-boosting features build into the CPUs. And Hyper-Threading technology that allows one CPU core to appear as two CPUs, by sharing ALUs etc.
While establish an accurate baseline for some upstream measurements (subj: Get rid of ndo_xmit_flush / commit 0b725a2ca61). I was starting to see too much variation in my trafgen measurements.
I created a rather-large oneliner, that I have converted into a script here: https://github.com/netoptimizer/network-testing/blob/master/bin/mon-ifpps
Which allowed me to get a picture of the accuracy of my measurements, and they are not accurate enough. (For more real stats like std-dev consider running these measurements through Rusty Russell's tool
- Disable all C states and P states.
- Disabling Hyper-Threading and power-management in BIOS helped the accuracy
- 10Gbit/s ixgbe ring-buffer cleanup interval also influenced accuracy
Reading +Jeremy Eder's blog post. It seems the best method for disabling these C and P states, and
keeping all CPUs in C0/C1 state is doing:
# tuned-adm profile latency-performance
I found the most stable ring-buffer cleanup interval for the ixgbe driver were 30 usecs. Configured by
# ethtool -C eth5 rx-usecs 30
Besides these tunings: my blogpost on "Basic tuning for network overload testing" should still be followed.
Generally I've started using the "profile latency-performance", but unless I need to measure some specific code change, I'm still using the ixgbe's default "dynamic" ring-buffer cleanup interval.
Details about the "ethtool -C" tuning is avail in blogpost "Pktgen for network overload testing".
The basic problem of using large MTU (usually 1500 bytes) size packets, is that the transmission delay itself, is enough to hide any improvement I'm making (e.g. a faster lookup function).
Transmission delay 1514 bytes (+20 bytes for Ethernet overhead) at 10Gbit/s is 1227 nanosec:
- ((bytes+wireoverhead)*8) / 10 Gbits = time-unit
- ((1500+14+20)*8)/((10000*10^6))*10^9 = 1227.20 ns
This means, if the network stack can generate (alloc/fill/free) a 1500 byte packet faster than every 1227ns, then it can utilize the bandwidth of the 10Gbit/s link fully. And yes, we can already do so. Thus, with 1500 bytes frame any stack performance improvement, will only be measurable with by a lower CPU utilization.
Let face it; the kernel have been optimized heavily for the last 20 years. Thus, the improvements we are able to come up with, is going to be on the nanosec scale.
For example I've found a faster way to clear the SKB, which saves 7 nanosec. Being able to measure this performance improvement were essential while developing this faster clearing.
Lets assume, the stack cost (alloc/fill/syscall/free) is 1200ns (thus faster than 1227ns), then a 7ns improvement will only be 0.58%, which I can only measure as a lower CPU utilization (as bandwidth limit have been reached), which in practice cannot be measured accurately enough.
By lowering the packet size, the transmission delay the stack cost (alloc/fill/syscall/free) can "hide-behind" is reduced. With the smallest packet size of 64 bytes, this is significantly reduced, to:
- ((64+20)*8)/((10000*10^6))*10^9 = 67.2ns
This basically exposes stacks cost, as its current cost is larger than 67.2ns. This can be used for getting some measurements that allow us to actually measure the improvement of the code changes we are making, even-though this "only" translates into reduced CPU usage with big frames (which translates into more processer time for your application).
In packet per sec (pps) this correspond to approx 14.8Mpps:
- 1sec/67.2ns => 1/(67.2/10^9) = 14,880,952 pps
- or directly from the packet size as:
- 10Gbit / (bytes*8) = (10000*10^6)/((64+20)*8) = 14,880,952 pps
Measuring packet per sec (PPS) instead of bandwidth, have another advantage. Instead of just comparing how many PPS improvement is seen, then instead translate the PPS into nanosec (between packets).
Comparing nanosec used before and after, will show us the nanosec saved by the given code change.
See, how I used it in this and this commit to document the actually improvement of the changes I made.
Update: For deducting the nanosec saved by a given code change, to be valid, usually requires isolating your test to utilize a single CPU.
Lets use the 14.8Mpps as an example of howto translate PPS to nanosec:
- 1sec / pps => (1/14880952*10^9) = 67.2ns
Extra: Just for the calculation exercise.
- How many packets per sec does 1227.20 ns correspond to:
- 1sec/1227.2ns => 1/(1227.2/10^9) = 814,863 pps
- Can also be calculated directly from the packet size as:
- 10Gbit / (bytes*8) = (10000*10^6)/((1514+20)*8) = 814,863 pps
Ignat Ignatov talked about physical formulas.
When I was planning to attend this talk, I thought it is going to be some sort
of symbolic formulas computation, possibly with an analysis of dimensions
of the physical quantities.
However, despite my (a bit long in the tooth) background in physics, I did not understand a word of it. Apparently, some sort of unification of physical formulas, not entirely unlike the periodic table in chemistry, was presented, with almost no comprehensible details and with scary words like co-homology and algebraic topology. The fact that half of the slides were in Russian, while irrelevant for me personally, probably did not help matters for the majority of the people in the audience. I did not expect any questions at the end of the talk, but there were at least two, so I was probably wrong about general level of understanding in the audience.
Laurent Dami talked about SQL::Abstract::FromQuery. He presented a query form of the Request Tracker and said that it is too complex - a premise many would agree with. The conclusion was that some more natural way to allow the user to specify complex queries is needed. Surprizingly, the answer to that was to use a formal grammar and make the user adhere to it. To me this sounds weird, but if one can find a non-empty set of users that would tolerate this, it may just work.
Denis Banovic talked about Docker, a virtualization container. I did not know much about Docker until this point, so it was useful to have someone to explain it to me.
The next talk was long, 50 minutes (as opposed to a somewhat standard for this conference 20 minutes) Peter "ribasushi" Rabbitson presented a crash-course in SQL syntax and concepts. It looked like a beginner-level introduction to SQL, but it became better and better as it progressed. I even learned a thing or two myself. ribasushi has a way of explaining rather complicated things concisely, understandably, and memorizably at the same time. Excellent talk.
Then there was a customary Subway sandwiches lunch.
Naim Shafiyev talked about network infrastructure automatization. Since this is closely related to what I do at my day job, I paid considerable attention to what he had to say. I did not hear anything new, but hopefuly the rest of the audience found the talk more useful. It did inspire me to submit a lightning talk though.
osfameron talked about immutable data structures in Perl and how to clone them with modifications, while making sure that the code does not look too ugly. Pretty standard stuff for functional languages, but pretty unusual in the land of Perl. The presentation was lively, with a lot of funny pictures and Donald duck examples.
The coffee break was followed by another session of lightning talks, preceeded by a give-away of a number of free books for the first-time YAPC attendees. Among the talks I remembered were SQLite virtual tables support in Perl by Laurent Dami, web-based database table editor by Simun Kodzoman, LeoNerd's presentation about XMPP replacement called Matrix, a Turing-complete (even if obfuscated) templating system by Jean-Baptiste Mazon of Sophia (sp!), and annoucements of Nordic Perl Workshop 2014 (Helsinki, November) and Nordic Perl Workshop 2015 (Oslo, May).
Again, I did not go to the end-of-the-day keynote.
As a side note, the wireless seemed to be substantially more flaky than yesterday, which has affected at least some lightning talk presenters.
When I came to the venue 15 minutes before the official start of the registration, people at the registration desk were busily cutting sheets of paper into attendees' badges. Finding my badge turned out to be a tad not trivial.
This conference is somewhat unusual not only because it is conducted over the weekend instead of in the middle of the week, but also because the keynotes for every day are pushed till the end, even after the daily lightning talks session.
The welcome talk from Marian was about practical things such as rooms locations, dinner, lunches, transportations and so on. Then I went on stage to declare the location of YAPC::Europe 2015 (which is Granada, Spain by the way). After that Jose Luis Martinez from Barcelona.pm did a short presentation of YAPC in Granada, and Diego Kuperman gave a little present from Granada to Sofia.
Mihai Pop of Cluj.pm presented a talk called "Perl Secret". It was basically a 20-minutes version of BooK's lightning talk about Perl secret operators, somewhat duluted by interspersing references to minions. It was entertaining.
The great Mark Overmeer talked about translation with context. He went beyond the usual example of multiple variants of plural values in some languages, and talked about solving localization problems related to gender and so on. The module solving these problems is Log::Report::Translate::Context. As always, great attention to details from Mark.
After lunch (sandwiches from Subway), Alex Balhatchet of Nestoria presented hurdles of geocoding, with solutions. I and my co-workers had encountered similar problems on a far smaller scale, so I could understand the pains, and had a great interest in hearing about the solutions.
Then I attended a very inspiring talk by Max Maischein from Frankfurt about using Perl as a DNLA remote and as a DNLA media server. I immediately felt the urge to play with the code he published and try to adapt it to my own TV at home. There was even a live demo of using DNLA to stream to Max's laptop a live stream of the talk provided by the conference organizers. And it even worked, mostly.
Ervin Ruci talked more about geocoding — this talk was partially touching the same problems Alex Balhatchet was talking about. Unfortunately, it was substantially less detailed, so I was somewhat underwhelmed by it. The presenter mentioned cool things like dealing with fuzzyness of the input data using hidden Markov models, but did not expand on them.
StrayTaoist described how to access raw data from space telescopes using (of course) Perl. Very lively talk. There was a lot of austronomy porn in here.
Luboŝ Kolouch from Czech Republic talked about automotive logistics, and how open source solutions work where proprietory solutions do not. The software needs to be reliable enough to make sure that it takes only 1.5 hours between the part order and its physical delivery to the factory.
After coffee break with more mingling the inimitable R Geoffrey Avery choir-mastered an hour of lightning talks. Most talks were somewhat "serious" today; I hope we see more "fun" ones in the next coming days.
Unfortunately, I missed the first keynote of the conference from Curtis "Ovid" Poe, so cannot really say anything about it.
Finally, we went to Restaurant Lebed for the conference dinner. The location is superb, there is a great view over a lake. The food was great, too. We also got to enjoy some ethnic Bulgarian music and dancing, not too much, and not too little.
Lots of cheers to Marian and the team of volunteers for organizing what so far turns out to be a great conference.
I’m glad to announce that One.com‘s public mirror service has begun mirroring Ceph‘s download section. Ceph is a distributed object store and file system, which scales elegantly and has excellent fault tolerance. Ceph has official mirrors in the Western US and the Netherlands, and a handful of community driven mirrors all over the world — now including this one in Denmark, well connected in Northern Europe. We welcome anyone using it to suit their needs. The Ceph mirror is available over HTTP here, and is also available over Rsync and FTP.
One.com run their mirror service both for operational independence and to be able to give something back to the open source software community. The service mirrors a number of open source projects, and more are added frequently. I’m lucky to be part of the team of mirror maintainers, and we’d love to hear from you if you have questions or ideas for the service.
Over the course of the last several months a lot of changes were made to the development process for openSUSE Factory. Meaning it’s no longer a highly experimental testing dump, but it’s now a viable rolling release distribution in its own right. You can read all about the details here. I installed openSUSE Factory in a virtual machine yesterday and it seems to run pretty great. Of course to really judge a rolling release distribution you need to run it for a sustained period of time.
No rolling release distribution will ever be my preferred day-to-day operating system, but nevertheless I’m pretty excited about the “new” openSUSE Factory. I think the changes will enable version whores and bleeding edge explorers to finally have a truly symbiotic relationship with the users who value productivity and predictability in their PC operating system.
Since I was already testing openSUSE Factory it was a great opportunity to finally get my feet wet with the new KDE Frameworks 5 and Qt5 based KDE Plasma 5 workspace, initially released about a month ago. Obviously it’s still lacking some features and polish, but it’s already usable for forgiving users who know what they’re doing and showing great promise.
My provider enabled 4G on my subscription and offered to send me a new SIM Card gratis. So now my Jolla is sporting 4G. Unfortunately it only took about 5-10 minutes of speed testing (peaking at 12 MB/s, averaging about 10 MB/s) to use all my available bandwidth for the month, so for the rest of August I’ve been speed dropped to 64 Kbps, but hey, it’s still 4G!
Who’d have ever thought they’d see the day that Stallman would do a presentation with accompanying slides? Well it happened, and I think this great use of slides helps him communicate more effectively. Watch the video and judge for yourselves (27 MB, 13 minutes).