On the 2014 freenet-meltdown

Update (2018-04-14): After 3 years of testing we could confidently decrease the minimum required connections to 6 (for 10 KiB/s upload speed) and still get good scaling thanks to the link length fix. This was released as Freenet build 1480. This yielded the same transfer speeds with lower CPU requirements and with much better scaling for high bandwidth nodes.

Update (2014-09-06): The meltdown is stopped and reversed. We implemented the link length fix and this solved an issue with the network structure we had for the last few years. We’re currently watching anxiously, whether the performance only comes back to the performance before the backdown or whether the lifetime actually gets much better. Watch the fetch-pull stats!

^ inserted one day ago: You see the meltdown starting in april and the improvement with the latest version: It’s back on at least the level before the meltdown.

^ 4 weeks ago inserted, 2 weeks ago accessed. If this goes above 0.6 starting 2014-09-19, the improvement could prove to be huge: It could give us much longer lifetimes of data in freenet.

Update (2014-07-23): The fetch-pull graphs look like we have oscillating here. This could mean that this is NOT an attack, but rather the direct effect of the KittyPorn patches: First the good connections get broken. This ruins the network. Then they can’t get any worse and the network recovers. Then they break again. This is still speculative. For an up to date plot, see fetchplots1.

Update (2014-05-22): The performance stats are much better again and the link-length distribution recovered. We might have been hit by an attack after all (which failed to take down freenet, but hurt pretty much). With some luck we’ll soon see a paper published with the evaluation of the attack and ways to mitigate it cleanly. (hint to computer scientists: we link to papers from the freenetproject.org website, so if you want to get a small boost for your chances of citation, send the link to your paper to devl@freenetproject.org)

Summary: There is a freenet patch floating around which claims to increase performance. The reality is (to our current-knowledge), that it breaks the network as soon as more than a few percent run it. And this is the case, which is why the network is almost completely broken right now. If you run that patch, please get rid of it!

Freenet is currently experiencing a meltdown, with extremely slow downloads, high connection churn and lifetimes for bigger files down to about a day. For a visualization, see the fetch-performance in the following graph and take note of the drop at the end. It nicely shows how a bad patch spread while more and more users installed it (hoping for better performance) and slowly killed the network. When that line goes below 50%, bigger files are dead approximately one day after being uploaded.

(thanks for these stats goes to fetchpull from digger3)

We suspect that patch, because the number of nodes reporting 100 or more connections in the anonymised probe-stats increased a lot over the past few weeks (this is only possible with a patched freenet) and the link-length-distribution almost completely lost a bump it had at 0.004, suggesting that freenet essentially reverted to random routing, while the number of nodes did not change significantly.

(thanks for these stats goes to probe stats which operhiem1 implemented in Google Summer of Code 2012)

We are working on creating a clean solution.

Freesites still work, because the SSK-queue did not get hammered, so if you are a freesite author, please inform your readers about the patch and ask them to get rid of it!

In case you use freenet and want information on that patch, please read the note from TheSeeker:

Information from TheSeeker in IRC (#freenet @ freenode)

Recently Kittyporn released an autopatcher-script: CHK@r6dUGAYs2No4lWT3DTkY2dIYdgA-eoBAcU~U-kLU-0I,hxGN5OTN4j~04YnCS4UTflMK4fpW2hfhB58CU1KNRAw,AAMC--8/FNAutoPatch-1.0_by_Kittyporn.rar

This increased usage of the patch by probably several hundred nodes, judging by the partial logs from the webserver that we have for fetches of the source tarball.

The script stupidly pulls the freenet source from freenetproject.org rather than say, github, or freenet. Really bad for anonymity, but good for tracking.

logs only go back a couple weeks, which is why they are incomplete, and we don't know the real number of people that have run it. hard to tell how much less the people that are cheating feel the effects of the whole network collapsing around them. surely can't be long before they too start complaining about speeds given the data retention issues it's causing.

NLM was supposed to fix all this shit. :|

modified nodes are flooding the network, creating broad backoff issues. this makes routing suffer, and avg path lengths increase, which reduces overall availability of bandwidth and more backoff and more misrouting. Death spiral until we hit some equilibrium that is roughly equal to random routing.

essentially what the broken NLM did. thankfully, it is only routing for bulk chk, so it'll still be possible to do some things if forced through the realtime queue... e.g. if we want to deploy an update, and have the constituent blocks actually get routed anywhere near the correct destination...

Additional comment

To do the math: a few hundred users easily equals 10% of the network. No wonder we have a meltdown.

and even worse, these few hundred users are likely the high-bandwidth folks with a huge number of connections.

Let’s assume that they each have 40 connections while the others have ~10. Every node connected to such an abusive node will essentially be blocked. That’s 100% of the nodes…

40 other nodes wrecked × 10% = ouch!