Anonymous DVCS in the Darknet.

Update 2024: Infocalypse is still recovering from Python 3 breakage. Most of it works again, but there may be rough edges left. Contributions to fix these are very welcome: hg.sr.ht/~arnebab/infocalypse or github.com/hyphanet/infocalypse.

This is a mirror of the documentation of the infocalypse extension for Mercurial written by djk - published here with his permission. It is licensed solely under the GPLv2 or later. The text is long. For concise information, use the second Link above (Freenet Development over Freenet).

Introduction

The Infocalypse 2.0 hg extension is an extension for Mercurial that allows you to create, publish and maintain incrementally updateable repositories in Freenet.

Your code is then hosted decentrally and anonymously, making it just as censorship-resistant as all other content in Freenet.

It works better than the other DVCS currently available for Freenet.

Most of the information you will find in this document can also be found in the extension's online help. i.e.:

hg help infocalypse

HOWTO: Infocalypse 2.0 hg extension

---

updated: 20090927

Note: Contains Freenet only links

Table of Contents

• Introduction
• Requirements
• Installation
• Updating
• Background
• Basic Usage
  • Generating a new private key
  • Creating a new repository
  • Pushing to a repository
  • Pulling from a repository
• Using FMS to send and receive update notifications
  • The update trust map
  • Reading other people's notifications
  • Posting your own notifications
  • Pulling an announced repository
• Reinserting and 'sponsoring' repositories
• Forking a repository onto a new USK
• Sharing private keys
• Inserting a freesite
• Risks
• Advocacy
• Source Code
• Fixes and version information
• Freenet-only links
• Contact

---

Requirements

The extension has the following dependencies:

• Freenet
  You can more information on Freenet here:
  
  http://freenetproject.org/ [HTTP Link!]

• Python
  I test on Python 2.5.4 and 2.6.1. Any 2.5.x or later version should work. Earlier versions may work.

  You probably won't have to worry about installing Python. It's included in the Windows binary Mercurial distributions and most *nix flavor OS's should have a reasonably up to date version of Python installed.

• Mercurial
  You can find more information on Mercurial here:
  
  http://mercurial-scm.org/ [HTTP Link!]

  Version 1.0.2 won't work.

  I use version 1.2.1 (x86 Gentoo) on a daily basis. Later versions should work.

  I've smoke tested 1.1.2 (on Ubuntu Jaunty Jackalope) and 1.3 (on Widows XP) without finding any problems.

• FMS
  Installation of the Freenet Messaging System (FMS) is optional but
  highly recommended. The hg fn-fmsread and hg fn-fmsnotify commands won't work without FMS. Without fn-fmsread it is extremely difficult to reliably detect repository updates.

  The official FMS freesite is here:

  USK@0npnMrqZNKRCRoGojZV93UNHCMN-6UU3rRSAmP6jNLE,~BG-edFtdCC1cSH4O3BWdeIYa8Sw5DfyrSV-TKdO5ec,AQACAAE/fms/106/
  
  

[TOC]

---

Installation

You checked the requirements and understandthe risks right?

Here are step-by-step instructions on how to install the extension.

• Download the bootstrap hg bundle:
  

  CHK@S~kAIr~UlpPu7mHNTQV0VlpZk-f~z0a71f7DlyPS0Do,IB-B5Hd7WePtvQuzaUGrVrozN8ibCaZBw3bQr2FvP5Y,AAIC--8/infocalypse2_1723a8de6e7c.hg
      

  You'll get a Potentially Dangerous Content warning from fproxy because the mime type isn't set. Choose 'Click here to force your browser to download the file to disk.'.

  I'll refer to the directory that you saved the bundle file to as DOWNLOAD_DIR.

• Create an empty directory where you want to install the extension.
  I'll refer to that directory as INSTALL_DIR in the
  rest of these instructions.

• Create an empty hg repository there. i.e.:
  

  cd INSTALL_DIR
  hg init
  

• Unbundle the bootstrap bundle into the new repository. i.e:
  

  hg pull DOWNLOAD_DIR/infocalypse2_1723a8de6e7c.hg
  hg update
  

• Edit the '[extensions]' section of your .hgrc/mercurial.ini
  file to point to the infocalypse directory in the unbundled source.

  # .hgrc/mercurial.ini snippet
  [extensions]
  infocalypse = INSTALL_DIR/infocalypse
  

  where INSTALL_DIR is the directory you unbundled into.

  If you don't known where to find/create your .hgrc/mercurial.ini file this link may be useful:
  
  http://www.selenic.com/mercurial/hgrc.5.html [HTTP Link!]

• Run fn-setup to create the config file and temp dir. i.e.
  

  hg fn-setup
     

  If you run your Freenet node on another machine or on a non-standard port you'll need to use the --fcphost and/or --fcpport parameters to set the FCP host and port respectively.

  By default fn-setup will write the configuration file for the extension (.infocalype on *nix, infocalypse.ini on Windows) into your home directory and also create a temp directory called infocalypse_tmp there.

  You can change the location of the temp directory by using the --tmpdir argument.

  If you want to put the config file in a different location set the cfg_file option in the [infocalypse] section of your .hgrc/mercurial.ini file before running fn-setup.

  Example .hgrc entry:
  # Snip, from .hgrc
  [infocalypse]
  cfg_file = /mnt/usbkey/s3kr1t/infocalypse.cfg

• Edit the fms_id and possibly fms_host/fms_port information in the.infocalyse/infocalypse.ini file. i.e.:
  
  

  # Example .infocalypse snippet
  fms_id = YOUR_FMS_ID
  
  fms_host = 127.0.0.1
  fms_port = 1119
  

  where YOUR_FMS_ID is the part of your fms id before the '@' sign.

  If you run FMS with the default settings on the same machine you are running
  Mercurial on you probably won't need to adjust the fcp_host or fcp_port.

  You can skip this step if you're not running fms.

• Read the latest know version of the extension's repository USK index from FMS.
  

  hg fn-fmsread -v
  

  You can skip this step if you're not running fms.

• Pull the latest changes to the extension from Freenet for the first time. Don't skip this step! i.e.:
  

  hg fn-pull --aggressive --debug --uri USK@kRM~jJVREwnN2qnA8R0Vt8HmpfRzBZ0j4rHC2cQ-0hw,2xcoQVdQLyqfTpF2DpkdUIbHFCeL4W~2X1phUYymnhM,AQACAAE/infocalypse.hgext.R1/41
  hg update
  

  You may have trouble finding the top key if you're not using fn-fmsread. Just keep retrying. If you know the index has increased, use the new index in the URI.

  After the first pull, you can update without the URI.

[TOC]

---

Updating

This extension is under active development. You should periodically update to get the latest bug fixes and new features.

Once you've installed the extension and pulled it for the first time, you can get updates by cd'ing into the initial INSTALL_DIRand typing:

hg fn-fmsread -vhg fn-pull --aggressive hg update

If you're not running FMS you can skip the fn-fmsread step. You may have trouble getting the top key. Just keep retrying.

If you're having trouble updating and you know the index has increased, use the full URI with the new index as above.

[TOC]

---

Background

Here's background information that's useful when using the extension.See the
Infocalypse 2.0 hg extension page on my freesite for a more detailed description of how the extension works.

Repositories are collections of hg bundle files

An Infocalypse repository is just a collection of hg bundle files which have been inserted into Freenet as CHKs and some metadata describing how to pull the bundles to reconstruct the repository that they represent. When you 'push' to an infocalypse repository a new bundleCHK is inserted with the changes since the last update. When you 'pull', only the CHKs for bundles for changesets not already in the local repository need to be fetched.

Repository USKs

The latest version of the repository's metadata is stored on a Freenet Updateable Subspace Key (USK) as a small binary file.

You'll notice that repository USKs end with a number without a trailing '/'. This is an important distinction. A repository USK is not a freesite. If you try to view one with fproxy you'll just get a 'Potentially Dangerous Content' warning. This is harmless, and ugly but unavoidable at the current time because of limitation in fproxy/FCP.

Repository top key redundancy

Repository USKs that end in *.R1/<number> are inserted redundantly, with a second USK insert done on *.R0/<number>. Top key redundancy makes it easier for other people to fetch your repository.

Inserting to a redundant repository USK makes the inserter more vulnerable to
correlation attacks. Don't use '.R1' USKs if you're worried about this.

Repository Hashes

Repository USKs can be long and cumbersome. A repository hash is the first 12 bytes of the Sha1 hash of the zero index version of a repository USK. e.g.:

SHA1( USK@kRM~jJVREwnN2qnA8R0Vt8HmpfRzBZ0j4rHC2cQ-0hw,2xcoQVdQLyqfTpF2DpkdUIbHFCeL4W~2X1phUYymnhM,AQACAAE/infocalypse.hgext.R1/0 )
  == 'be68e8feccdd'

You can get the repository hash for a repository USK using:

hg fn-info

from a directory the repository USK has been fn-pull'd into.

You can get the hashes of repositories that other people have announced via fms with:

hg fn-fmsread --listall

Repository hashes are used in the fms update trust map.

The default private key

When you run fn-setup, it creates a default SSK private key, which it stores inthe default_private_key parameter in your .infocalypse/infocalypse.ini file.

You can edit the config file to substitute any valid SSK private key you want.

If you specify an Insert URI without the key part for an infocalypse command the default private key is filled in for you. i.e

hg fn-create --uri USK@/test.R1/0

Inserts the local hg repository into a new USK in Freenet, using the private key in your config file.

USK <--> Directory mappings

The extension's commands 'remember' the insert and request repository USKs they were last run with when run again from the same directory.

This makes it unnecessary to retype cumbersome repository USK values once a repository has been successfully pulled or pushed from a directory.

Aggressive top key searching

fn-pull and fn-push have an --aggressive command line argument which causes them to search harder for the latest request URI.

This can be slow, especially if the USK index is much lower than the latest index in Freenet.

You will need to use it if you're not using FMS update notifications.

[TOC]

---

Basic Usage

Here are examples of basic commands.

Generating a new private key

You can generate an new private key with:

hg fn-genkey

This has no effect on the stored default private key.

Make sure to change the 'SSK' in the InsertURI to 'USK' when supplying the insert URI on the command line.

Creating a new repository

hg fn-create --uri USK@/test.R1/0

Inserts the local hg repository into a new USK in Freenet, using the privatekey in your config file. You can use a full insert URI value if you want.

If you see an "update -- Bundle too big to salt!" warning message when you run this command you should consider running
fn-reinsert --level 4.

Pushing to a repository

hg fn-push --uri USK@/test.R1/0

Pushes incremental changes from the local directory into an existing Infocalypse repository.

The <keypart>/test.R1/0 repository must already exist in Freenet.In the example above the default private key is used. You could have specified a full Insert URI. The URI must end in a number but the value doesn't matter because fn-push searches for the latest unused index.

You can ommit the --uri argument whenyou run from the same directory the fn-create (or a previous fn-push)was run from.

Pulling from a repository

hg fn-pull --uri <request uri>

pulls from an Infocalypse repository in Freenet intothe local repository.
Here's an example with a fully specified uri.

You can ommit the --uri argument whenyou run from the same directory a previous fn-pull was successfully run from.

For maximum reliability use the --aggressive argument.

[TOC]

---

Using FMS to send and receive update notifications

The extension can send and receive repository update notifications via FMS. It is highly recommended that you
setup this feature.

The update trust map

There's a trust map in the .infocalypse/infocalypse.ini config file which determines which fms ids can update the index values for which repositories. It is purely local and completely separate from the trust values which appear in the FMS web of trust.

The format is:
<number> = <fms_id>|<usk_hash0>|<usk_hash1>| ... |<usk_hashn>

The number value must be unique, but is ignored.

The fms_id values are the full FMS ids that you are trusting to update the repositories with the listed hashes.

The usk_hash* values are repository hashes.

Here's an example trust map config entry:

# Example .infocalypse snippet
[fmsread_trust_map]
1 = test0@adnT6a9yUSEWe5p8J-O1i8rJCDPqccY~dVvAmtMuC9Q|55833b3e6419
0 = djk@isFiaD04zgAgnrEC5XJt1i4IE7AkNPqhBG5bONi6Yks|be68e8feccdd|5582404a9124
2 = test1@SH1BCHw-47oD9~B56SkijxfE35M9XUvqXLX1aYyZNyA|fab7c8bd2fc3

You must update the trust map to enable index updating for repos other than the one this code lives in (be68e8feccdd). You can edit the config file directly if you want.

However, the easiest way to update the trust map is by using the--trust and --untrust options on fn-fmsread.

For example to trust falafel@IxVqeqM0LyYdTmYAf5z49SJZUxr7NtQkOqVYG0hvITwto notify you about changes to the repository with repo hash 2220b02cf7ee,type:

hg fn-fmsread --trust --hash 2220b02cf7ee --fmsid falafel@IxVqeqM0LyYdTmYAf5z49SJZUxr7NtQkOqVYG0hvITw

And to stop trusting that FMS id for updates to 2220b02cf7ee, you would type:

hg fn-fmsread --untrust --hash 2220b02cf7ee --fmsid falafel@IxVqeqM0LyYdTmYAf5z49SJZUxr7NtQkOqVYG0hvITw

To show the trust map type:

hg fn-fmsread --showtrust

Reading other people's notifications

hg fn-fmsread -v

Will read update notifications for all the repos in the trust map and locally cache the new latest index values. If you run with -vit prints a message when updates are available which weren't used because the sender(s) weren't in the trust map.

hg fn-fmsread --list

Displays announced repositories from fms ids that appear inthe trust map.

hg fn-fmsread --listall

Displays all announced repositories including ones from unknown fms ids.

Pulling an announced repository

You can use the --hash option with fn-pull to pull any repository you see in the fn-read --list or fn-read --listall lists.

For example to pull the latest version of the infocalypse extension code, cd to an empty directory and type:

hg inithg fn-pull --hash be68e8feccdd --aggressive

Posting your own notifications

hg fn-fmsnotify -v

Posts an update notification for the current repository to fms.

You MUST set the fms_id value in the config fileto your fms id for this to work.

Use --dryrun to double check before sending the actual fms message.

Use --announce at least once if you want your USK to show up in the fmsread --listall list.

By default notifications are written to and read from the infocalypse.notify fms group.

The read and write groups can be changed by editing the following variables in the config file:

fmsnotify_group = <group>
fmsread_groups = <group0>[|<group1>|...]

fms can have pretty high latency. Be patient. It may take hours (sometimes a day!) for your notification to appear. Don't send lots of redundant notifications.

[TOC]

---

Reinserting and 'sponsoring' repositories

hg fn-reinsert

will re-insert the bundles for the repository that was last pulled into the directory.

The exact behavior is determined by the level argument.

level:

• 1 - re-inserts the top key(s)
• 2 - re-inserts the top keys(s), graphs(s) and the most recent update.
• 3 - re-inserts the top keys(s), graphs(s) and all keys required to bootstrap the repo.

  This is the default level.

• 4 - adds redundancy for big (>7Mb) updates.
• 5 - re-inserts existing redundant big updates.

Levels 1 and 4 require that you have the privatekey for the repository. For other levels, the top key insert is skipped if you don't have the private key.

DO NOT use fn-reinsert if you're concerned about
correlation attacks. The risk is on the order of re-inserting a freesite, but may be worse if you use redundant(i.e. USK@<line noise>/name.R1/0) top keys.

[TOC]

---

Forking a repository onto a new USK

hg fn-copy --inserturi USK@/name_for_my_copy.R1/0

copies the Infocalypse repository which was fn-pull'd intothe local directory onto a new repository USK under your default private key. You can use a full insert URI if you want.

This only requires copying the top key data (a maximum of 2 SSK inserts).

[TOC]

---

Sharing private keys

It is possible for multiple people to collaborate anonymously over Freenet by sharing the private key to a single Infocalypse repository.

The FreeFAQ is an example of this technique.

Here are some things to keep in mind when sharing private keys.

• There is no (explict) key revocation in Freenet
  
  If you decide to share keys, you should generate a special key on a per repo basis with fn-genkey. There is no way to revoke a private key once it has been shared. This could be mitigated with an ad-hoc convention. e.g. if I find any file named USK@<public_key>/revoked.txt, I stop using the key.
• Non-atomic top key inserts
  
  Occasionally, you might end up overwriting someone elses commits because the FCP insert of the repo top key isn't atomic. I think you should be able to merge and re fn-push to resolve this. You can fn-pull a specific version of the repo by specify the full URI including the version number with --uri and including the --nosearch option.
• All contributors should be in the fn-fmsread trust map

[TOC]

---

Inserting a freesite

hg fn-putsite --index <n>

inserts a freesite based on the configuration inthe freesite.cfg file in the root of the repository.

Use:

hg fn-putsite --createconfig

to create a basic freesite.cfg file that you can modify. Look at the comments in it for an explanation of the supported parameters.

The default freesite.cfg file inserts using the same private key as the repo and a site name of 'default'. Editing the name is highly recommended.

You can use --key CHK@ to insert a test version of the site to a CHK key before writing to the USK.

Limitations:

• You MUST have fn-pushed the repo at least once in order to insert using the repo's private key. If you haven't fn-push'd you'll see this error: "You don't have the insert URI for this repo. Supply a private key with --key or fn-push the repo."
• Inserts all files in the site_dir directory in the freesite.cfg file. Run with --dryrun to make
  sure that you aren't going to insert stuff you don't want too.
• You must manually specify the USK edition you want to insert on. You will get a collision error
  if you specify an index that was already inserted.
• Don't use this for big sites. It should be fine for notes on your project. If you have lots of images
  or big binary files use a tool like jSite instead.
• Don't modify site files while the fn-putsite is running.

[TOC]

---

Risks

I don't believe that using this extension is significantly more dangerous that using any other piece of Freenet client code, but here is a list of the risks which come to mind:

• Freenet is beta software
  The authors of Freenet don't pretend to guarantee that it is free of bugs that could that could compromise your anonymity or worse.

  While written in Java, Freenet loads native code via JNI (FEC codecs, bigint stuff, wrapper, etc.) that makes it vulnerable to the same kinds of attacks as any other C/C++ code.

• FMS == anonymous software
  FMS is published anonymously on Freenet and it is written in C++ with dependencies on large libraries which could contain security defects.

  I personally build FMS from source and run it in a chroot jail.

  Somedude, the author of FMS, seems like a reputable guy and has conducted himself as such for more than a year.

• correlation attacks
  There is a concern that any system which inserts keys that can be predicted ahead of time could allow an attacker with control over many nodes in the network to eventually find the IP of your node.

  Any system which has this property is vulnerable. e.g. fproxy Freesite insertion,Freetalk, FMS, FLIP. This extension's optional use of
  redundant top keys may make it particularly vulnerable. If you are concerned don't use '.R1' keys.

  Running your node in pure darknet mode with trusted peers may somewhat reduce the risk of correlation attacks.

• Bugs in my code, Mercurial or Python
  I do my best but no one's perfect.

  There are lots of eyes over the Mercurial and Python source.

[TOC]

---

Advocacy

Here are some reasons why I think the Infocalypse 2.0 hg extension is better than
pyFreenetHg and
egit-freenet:

• Incremental

  You only need to insert/retrieve what has actually changed. Changes of up to 32kof compressed deltas can be fetched in as little as one SSK fetch and one CHK fetch.

• Redundant

  The top level SSK and the CHK with the representation of the repository state are inserted redundantly so there are no 'critical path' keys. Updates of up to ~= 7Mbare inserted redundantly by cloning the splitfile metadata at the cost of a single32k CHK insert.

• Re-insertable

  Anyone can re-insert all repository data except for the top level SSKs with a simple command (hg fn-reinsert). The repository owner can re-insert the top levelSSKs as well.

• Automatic rollups

  Older changes are automatically 'rolled up' into large splitfiles, such that the entire repository can almost always be fetched in 4 CHK fetches or less.

• Fails explictly

  REDFLAG DCI

[TOC]

---

Source Code

The authoritative repository for the extension's code is hosted in Freenet:

hg inithg fn-fmsread -vhg fn-pull --aggressive --debug --uri USK@kRM~jJVREwnN2qnA8R0Vt8HmpfRzBZ0j4rHC2cQ-0hw,2xcoQVdQLyqfTpF2DpkdUIbHFCeL4W~2X1phUYymnhM,AQACAAE/infocalypse.hgext.R1/41hg update

It is also mirrored on bitbucket.org:

hg clone http://bitbucket.org/dkarbott/infocalypse_hgext/

[TOC]

---

Fixes and version information

• hg version: c51dc4b0d282
  
  Fixed abort: <bundle_file> not found! problem on fn-pull when hg-git plugin was loaded.
• hg version: 0c5ce9e6b3b4
  
  Fixed intermittent stall when bootstrapping from an empty repo.
• hg version: 7f39b20500f0
  
  Fixed bug that kept fn-pull --hash from updating the initial USK index.
• hg version: 7b10fa400be1
  
  Added fn-fmsread --trust and --untrust and fn-pull --hash support.
  
  
  fn-pull --hash isn't really usable until 7f39b20500f0
• hg version: ea6efac8e3f6
  
  Fixed a bug that was causing the berkwood binary 1.3 Mercurial distribution
  (http://mercurial.berkwood.com/binaries/Mercurial-1.3.exe [HTTP Link!]) not to work.

[TOC]

---

Freenet-only links

This document is meant to inserted into Freenet.

It contains links (starting with 'CHK@' and 'USK@')to Freenet keys that will only work from within fproxy [HTTP link!].

You can find reasonably up to date version of this document on my freesite:

USK@-bk9znYylSCOEDuSWAvo5m72nUeMxKkDmH3nIqAeI-0,qfu5H3FZsZ-5rfNBY-jQHS5Ke7AT2PtJWd13IrPZjcg,AQACAAE/feral_codewright/15/infocalypse_howto.html

[TOC]

---

Contact

FMS:
djk@isFiaD04zgAgnrEC5XJt1i4IE7AkNPqhBG5bONi6Yks

I lurk on the freenet and fms boards.

If you really need to you can email me at d kar bott at com cast dot net but I prefer FMS.

freesite:
USK@-bk9znYylSCOEDuSWAvo5m72nUeMxKkDmH3nIqAeI-0,qfu5H3FZsZ-5rfNBY-jQHS5Ke7AT2PtJWd13IrPZjcg,AQACAAE/feral_codewright/15/

[TOC]