Babcore: Emacs Customizations everyone should have

C-g: abort

Table of Contents

• 1 Intro
• 2 Package Header
• 3 Feature Gems
  • 3.1 package.el, full setup
  • 3.2 Flymake
  • 3.3 auto-complete
  • 3.4 ido
  • 3.5 printing
  • 3.6 outlining everywhere
  • 3.7 Syntax highlighting
  • 3.8 org and babel
  • 3.9 Nice line wrapping
  • 3.10 goto-chg
  • 3.11 flyspell
  • 3.12 control-lock
  • 3.13 Basic key chords
  • 3.14 X11 tricks
    • 3.14.1 frame-to-front
    • 3.14.2 urgency hint
    • 3.14.3 fullscreen mode
    • 3.14.4 default key bindings
  • 3.15 Insert unicode characters
  • 3.16 Highlight TODO and FIXME in comments
  • 3.17 Save macros as functions
  • 3.18 Transparent GnuPG encryption
  • 3.19 Colored shell commands
  • 3.20 Save backups in ~/.local/share/emacs-saves
  • 3.21 Basic persistency
    • 3.21.1 saveplace
    • 3.21.2 recentf
    • 3.21.3 savehist
    • 3.21.4 desktop globals
  • 3.22 use the system clipboard
  • 3.23 Add license headers automatically
  • 3.24 finish up
• 4 Summary

;; Copyright (C) 2013 Arne Babenhauserheide

;; Author: Arne Babenhauserheide (and various others in Emacswiki and elsewhere).
;; Maintainer: Arne Babenhauserheide
;; Created 03 April 2013
;; Version: 0.0.2
;; Version Keywords: core configuration

;; This program is free software; you can redistribute it and/or
;; modify it under the terms of the GNU General Public License
;; as published by the Free Software Foundation; either version 3
;; of the License, or (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program. If not, see <http://www.gnu.org/licenses/>.

;;; Commentary:
;; Quick Start / installation:
;; 1. Download this file and put it next to other files Emacs includes
;; 2. Add this to you .emacs file and restart emacs:
;;      (require 'babcore)
;;
;; Use Case: Use a common core configuration so you can avoid the
;;   tedious act of gathering all the basic stuff over the years and
;;   can instead concentrate on the really cool new stuff Emacs offers
;;   you.
;;
;; Todo:
;;

;;; Change Log:
;; 2013-04-03 - Minor adjustments
;; 2013-02-29 - Initial release

;;; Code:

;; Convenient package handling in emacs

(require 'package)
;; use packages from marmalade
(add-to-list 'package-archives '("marmalade" . "http://marmalade-repo.org/packages/"))
;; and the old elpa repo
(add-to-list 'package-archives '("elpa-old" . "http://tromey.com/elpa/"))
;; and automatically parsed versiontracking repositories.
(add-to-list 'package-archives '("melpa" . "http://melpa.milkbox.net/packages/"))

;; Make sure a package is installed
(defun package-require (package)
  "Install a PACKAGE unless it is already installed 
or a feature with the same name is already active.

Usage: (package-require 'package)"
  ; try to activate the package with at least version 0.
  (package-activate package '(0))
  ; try to just require the package. Maybe the user has it in his local config
  (condition-case nil
      (require package)
    ; if we cannot require it, it does not exist, yet. So install it.
    (error (package-install package))))

;; Initialize installed packages
(package-initialize)  
;; package init not needed, since it is done anyway in emacs 24 after reading the init
;; but we have to load the list of available packages
(package-refresh-contents)

;; Flymake: On the fly syntax checking

; stronger error display
(defface flymake-message-face
  '((((class color) (background light)) (:foreground "#b2dfff"))
    (((class color) (background dark))  (:foreground "#b2dfff")))
  "Flymake message face")

; show the flymake errors in the minibuffer
(package-require 'flymake-cursor)  

;; Inline auto completion and suggestions
(package-require 'auto-complete)

; use ido mode for file and buffer Completion when switching buffers
(require 'ido)
(ido-mode t)

;; Convenient printing
(require 'printing)
(pr-update-menus t)
; make sure we use localhost as cups server
(setenv "CUPS_SERVER" "localhost")
(package-require 'cups)

; use allout minor mode to have outlining everywhere.
(allout-mode)

; syntax highlighting everywhere
(global-font-lock-mode 1)

; Activate org-mode
(require 'org)
; and some more org stuff

; http://orgmode.org/guide/Activation.html#Activation

; The following lines are always needed.  Choose your own keys.
(add-to-list 'auto-mode-alist '("\\.org\\'" . org-mode))

; And add babel inline code execution
; babel, for executing code in org-mode.
(org-babel-do-load-languages
 'org-babel-load-languages
 ; load all language marked with (lang . t).
 '((C . t)
   (R . t)
   (asymptote)
   (awk)
   (calc)
   (clojure)
   (comint)
   (css)
   (ditaa . t)
   (dot . t)
   (emacs-lisp . t)
   (fortran)
   (gnuplot . t)
   (haskell)
   (io)
   (java)
   (js)
   (latex)
   (ledger)
   (lilypond)
   (lisp)
   (matlab)
   (maxima)
   (mscgen)
   (ocaml)
   (octave)
   (org . t)
   (perl)
   (picolisp)
   (plantuml)
   (python . t)
   (ref)
   (ruby)
   (sass)
   (scala)
   (scheme)
   (screen)
   (sh . t)
   (shen)
   (sql)
   (sqlite)))

; Add proper word wrapping
(global-visual-line-mode t)

; go to the last change
(package-require 'goto-chg)
(global-set-key [(control .)] 'goto-last-change)
; M-. can conflict with etags tag search. But C-. can get overwritten
; by flyspell-auto-correct-word. And goto-last-change needs a really
; fast key.
(global-set-key [(meta .)] 'goto-last-change)

; Make german umlauts work.
(setq locale-coding-system 'utf-8)
(set-terminal-coding-system 'utf-8)
(set-keyboard-coding-system 'utf-8)
(set-selection-coding-system 'utf-8)
(prefer-coding-system 'utf-8)

;aspell und flyspell
(setq-default ispell-program-name "aspell")

;make aspell faster but less correctly
(setq ispell-extra-args '("--sug-mode=ultra" "-w" "äöüÄÖÜßñ"))
(setq ispell-list-command "list")

; control-lock-mode, so we can enter a vi style command-mode with standard emacs keys.
(package-require 'control-lock)
; also bind M-ü and M-ä to toggling control lock.
(global-set-key (kbd "M-ü") 'control-lock-toggle)
(global-set-key (kbd "C-ü") 'control-lock-toggle)
(global-set-key (kbd "M-ä") 'control-lock-toggle)
(global-set-key (kbd "C-ä") 'control-lock-toggle)
(global-set-key (kbd "C-z") 'control-lock-toggle)

; use key chords invoke commands
(package-require 'key-chord)
(key-chord-mode 1)
; buffer actions
(key-chord-define-global "vg"     'eval-region)
(key-chord-define-global "vb"     'eval-buffer)
(key-chord-define-global "cy"     'yank-pop)
(key-chord-define-global "cg"     "\C-c\C-c")
; frame actions
(key-chord-define-global "xo"     'other-window);
(key-chord-define-global "x1"     'delete-other-windows)
(key-chord-define-global "x0"     'delete-window)
(defun kill-this-buffer-if-not-modified ()
  (interactive)
  ; taken from menu-bar.el
  (if (menu-bar-non-minibuffer-window-p)
      (kill-buffer-if-not-modified (current-buffer))
    (abort-recursive-edit)))
(key-chord-define-global "xk"     'kill-this-buffer-if-not-modified)
; file actions
(key-chord-define-global "bf"     'ido-switch-buffer)
(key-chord-define-global "cf"     'ido-find-file)
(key-chord-define-global "vc"     'vc-next-action)

emacsclient -e "(show-frame)"

(defun show-frame (&optional frame)
  "Show the current Emacs frame or the FRAME given as argument.

And make sure that it really shows up!"
  (raise-frame)
  ; yes, you have to call this twice. Don’t ask me why…
  ; select-frame-set-input-focus calls x-focus-frame and does a bit of
  ; additional magic.
  (select-frame-set-input-focus (selected-frame))
  (select-frame-set-input-focus (selected-frame)))

;; let emacs blink when something interesting happens.
;; in KDE this marks the active Emacs icon in the tray.
(defun x-urgency-hint (frame arg &optional source)
  "Set the x-urgency hint for the frame to arg: 

- If arg is nil, unset the urgency.
- If arg is any other value, set the urgency.

If you unset the urgency, you still have to visit the frame to make the urgency setting disappear (at least in KDE)."
  (let* ((wm-hints (append (x-window-property 
                            "WM_HINTS" frame "WM_HINTS" 
                            source nil t) nil))
         (flags (car wm-hints)))
    ; (message flags)
    (setcar wm-hints
            (if arg
                (logior flags #x00000100)
              (logand flags #x1ffffeff)))
    (x-change-window-property "WM_HINTS" wm-hints frame "WM_HINTS" 32 t)))

(defun x-urgent (&optional arg)
  "Mark the current emacs frame as requiring urgent attention. 

With a prefix argument which does not equal a boolean value of nil, remove the urgency flag (which might or might not change display, depending on the window manager)."
  (interactive "P")
  (let (frame (car (car (cdr (current-frame-configuration)))))
  (x-urgency-hint frame (not arg))))  

; fullscreen, taken from http://www.emacswiki.org/emacs/FullScreen#toc26
; should work for X und OSX with emacs 23.x (TODO find minimum version).
; for windows it uses (w32-send-sys-command #xf030) (#xf030 == 61488)
(defvar babcore-fullscreen-p t "Check if fullscreen is on or off")
(setq babcore-stored-frame-width nil)
(setq babcore-stored-frame-height nil)

(defun babcore-non-fullscreen ()
  (interactive)
  (if (fboundp 'w32-send-sys-command)
      ;; WM_SYSCOMMAND restore #xf120
      (w32-send-sys-command 61728)
    (progn (set-frame-parameter nil 'width 
                                (if babcore-stored-frame-width
                                    babcore-stored-frame-width 82))
           (set-frame-parameter nil 'height
                                (if babcore-stored-frame-height 
                                    babcore-stored-frame-height 42))
           (set-frame-parameter nil 'fullscreen nil))))

(defun babcore-fullscreen ()
  (interactive)
  (setq babcore-stored-frame-width (frame-width))
  (setq babcore-stored-frame-height (frame-height))
  (if (fboundp 'w32-send-sys-command)
      ;; WM_SYSCOMMAND maximaze #xf030
      (w32-send-sys-command 61488)
    (set-frame-parameter nil 'fullscreen 'fullboth)))

(defun toggle-fullscreen ()
  (interactive)
  (setq babcore-fullscreen-p (not babcore-fullscreen-p))
  (if babcore-fullscreen-p
      (babcore-non-fullscreen)
    (babcore-fullscreen)))

(global-set-key [f11] 'toggle-fullscreen)

; Default KDE keybindings to make emacs nicer integrated into KDE. 

; can treat C-m as its own mapping.
; (define-key input-decode-map "\C-m" [?\C-1])

(defun revert-buffer-preserve-modes ()
  (interactive)
  (revert-buffer t nil t))

; C-m shows/hides the menu bar - thanks to http://stackoverflow.com/questions/2298811/how-to-turn-off-alternative-enter-with-ctrlm-in-linux
; f5 reloads
(defconst kde-default-keys-minor-mode-map
  (let ((map (make-sparse-keymap)))
    (set-keymap-parent map text-mode-map)
    (define-key map [f5] 'revert-buffer-preserve-modes)
    (define-key map [?\C-1] 'menu-bar-mode)
    (define-key map [?\C-+] 'text-scale-increase)
    (define-key map [?\C--] 'text-scale-decrease) ; shadows 'negative-argument which is also available via M-- and C-M--, though.
    (define-key map [C-kp-add] 'text-scale-increase)
    (define-key map [C-kp-subtract] 'text-scale-decrease)
    map)
  "Keymap for `kde-default-keys-minor-mode'.")

;; Minor mode for keypad control
(define-minor-mode kde-default-keys-minor-mode
  "Adds some default KDE keybindings"
  :global t
  :init-value t
  :lighter ""
  :keymap 'kde-default-keys-minor-mode-map
  )

M-x ucs-insert

;; Highlight TODO and FIXME in comments 
(package-require 'fic-ext-mode)
(defun add-something-to-mode-hooks (mode-list something)
  "helper function to add a callback to multiple hooks"
  (dolist (mode mode-list)
    (add-hook (intern (concat (symbol-name mode) "-mode-hook")) something)))

(add-something-to-mode-hooks '(c++ tcl emacs-lisp python text markdown latex) 'fic-ext-mode)

; save the current macro as reusable function.
(defun save-current-kbd-macro-to-dot-emacs (name)
  "Save the current macro as named function definition inside
your initialization file so you can reuse it anytime in the
future."
  (interactive "SSave Macro as: ")
  (name-last-kbd-macro name)
  (save-excursion 
    (find-file-literally user-init-file)
    (goto-char (point-max))
    (insert "\n\n;; Saved macro\n")
    (insert-kbd-macro name)
    (insert "\n")))

; Activate transparent GnuPG encryption.
(require 'epa-file)
(epa-file-enable)

; colored shell commands via C-!
(add-hook 'shell-mode-hook 'ansi-color-for-comint-mode-on)
(defun babcore-shell-execute(cmd)
  "Execute a shell command in an interactive shell buffer."
   (interactive "sShell command: ")
   (shell (get-buffer-create "*shell-commands-buf*"))
   (process-send-string (get-buffer-process "*shell-commands-buf*") (concat cmd "\n")))
(global-set-key (kbd "C-!") 'babcore-shell-execute)

(setq backup-by-copying t      ; don't clobber symlinks
      backup-directory-alist
      '(("." . "~/.local/share/emacs-saves"))    ; don't litter my fs tree
      delete-old-versions t
      kept-new-versions 6
      kept-old-versions 2
      version-control t)       ; use versioned backups

; save the place in files
(require 'saveplace)
(setq-default save-place t)

; show recent files
(package-require 'recentf)
(recentf-mode 1)
(setq recentf-max-menu-items 1000)

; save minibuffer history
(require 'savehist)
(savehist-mode t)

;; save registers and open files over restarts,
;; thanks to http://www.xsteve.at/prg/emacs/power-user-tips.html
;; save a list of open files in ~/.emacs.desktop
;; save the desktop file automatically if it already exists
;(setq desktop-save 'if-exists)
;(desktop-save-mode 1)

;; ;; save a bunch of variables to the desktop file
;; ;; for lists specify the len of the maximal saved data also
;; (setq desktop-globals-to-save
;;       (append '((extended-command-history . 300)
;;                 (file-name-history        . 100)
;;                 (grep-history             . 30)
;;                 (compile-history          . 30)
;;                 (minibuffer-history       . 5000)
;;                 (query-replace-history    . 60)
;;                 (read-expression-history  . 60)
;;                 (regexp-history           . 60)
;;                 (regexp-search-ring       . 20)
;;                 (search-ring              . 2000)
;;                 (shell-command-history    . 50)
;;                 tags-file-name
;;                 register-alist)))

;; ;; restore only 5 buffers at once and the rest lazily
;; (setq desktop-restore-eager 5)

; maybe nicer: http://github.com/doomvox/desktop-recover

; Use the system clipboard
(setq x-select-enable-clipboard t)

(package-require 'legalese)

(provide 'babcore)
;;; babcore.el ends here  

Note: As almost everything on this page, this text and code is available under the GPLv3 or later.

Custom link completion for org-mode in 25 lines (emacs)

Update (2013-01-23): The new org-mode removed (org-make-link), so I replaced it with (concat) and uploaded a new example-file: org-custom-link-completion.el.
Happy Hacking!

1 Intro

I recently set up custom completion for two of my custom link types in Emacs org-mode. When I wrote on identi.ca about that, Greg Tucker-Kellog said that he’d like to see that. So I decided, I’d publish my code.

The link types I regularly need are papers (PDFs of research papers I take notes about) and bib (the bibtex entries for the papers). The following are my custom link definitions :

(setq org-link-abbrev-alist
      '(("bib" . "~/Dokumente/Uni/Doktorarbeit-inverse-co2-ch4/aufschriebe/ref.bib::%s")
       ("notes" . "~/Dokumente/Uni/Doktorarbeit-inverse-co2-ch4/aufschriebe/papers.org::#%s")
       ("papers" . "~/Dokumente/Uni/Doktorarbeit-inverse-co2-ch4/aufschriebe/papers/%s.pdf")))

For some weeks I had copied the info into the links by hand. Thus an entry about a paper looks like the following.

* Title [[bib:identifier]] [[papers:name_without_suffix]]

This already suffices to be able to click the links for opening the PDF or showing the bibtex entry. Entering the links was quite inconvenient, though.

(defun string-replace (this withthat in)
  "replace THIS with WITHTHAT' in the string IN"
  (with-temp-buffer
    (insert in)
    (goto-char (point-min))
    (replace-string this withthat)
    (buffer-substring (point-min) (point-max))))

<<string-replace>>
(defun org-papers-complete-link (&optional arg)
  "Create a papers link using completion."
  (let (file link)
       (setq file (read-file-name "papers: " "papers/"))
       <<cleanup-link>>
    link))

(let ((pwd (file-name-as-directory (expand-file-name ".")))
  (pwd1 (file-name-as-directory (abbreviate-file-name
                 (expand-file-name ".")))))
  (setq file (string-replace "papers/" "" file))
  (setq file (string-replace pwd "" (string-replace pwd1 "" file)))
  (setq file (string-replace ".pdf" "" file))
  (setq link (concat "papers:" file)))

<<reftex-setup>>
(defun org-bib-complete-link (&optional arg)
  "Create a bibtex link using reftex autocompletion."
  (concat "bib:" (reftex-do-citation nil t nil)))

(defun org-mode-reftex-setup ()
  (interactive)
  (and (buffer-file-name) (file-exists-p (buffer-file-name))
       (progn
        ; Reftex should use the org file as master file. See C-h v TeX-master for infos.
        (setq TeX-master t)
        (turn-on-reftex)
        ; don’t ask for the tex master on every start.
        (reftex-parse-all)
        ;add a custom reftex cite format to insert links
        (reftex-set-cite-format
         '((?b . "[[bib:%l][%l-bib]]")
           (?n . "[[notes:%l][%l-notes]]")
           (?p . "[[papers:%l][%l-paper]]")
           (?t . "%t")
           (?h . "** %t\n:PROPERTIES:\n:Custom_ID: %l\n:END:\n[[papers:%l][%l-paper]]")))))
  (define-key org-mode-map (kbd "C-c )") 'reftex-citation)
  (define-key org-mode-map (kbd "C-c (") 'org-mode-reftex-search))

(add-hook 'org-mode-hook 'org-mode-reftex-setup)

<<org-papers-complete-link>>
<<org-bib-complete-link>>

Easily converting ris-citations to bibtex with emacs and bibutils

The problem

Nature only gives me ris-formatted citations, but I use bibtex.

Also ris is far from human readable.

The background

ris can be reformatted to bibtext, but doing that manually disturbs my workflow when getting references while taking note about a paper in emacs.

I tend to search online for references, often just using google scholar, so when I find a ris reference, the first data I get for the ris-citation is a link.

The solution

Making it possible

bibutils¹ can convert ris to an intermediate xml format and then convert that to bibtex.

wget -O reference.ris RIS_URL
cat reference.ris | ris2xml | xml2bib >> ref.bib

This solves the problem, but it is not convenient, because I have to switch to the terminal, download the file, convert it and append the result to my bibtex file.

Making it convenient

With the first step, getting the ris-citation is quite inconvenient. I need 3 steps just for getting a citation.

But those steps are always the same, and since I use Emacs, I can automate and integrate them very easily. So I created a simple function in emacs, which takes the url of a ris citation, converts it to bibtex and appends the result to my local bibtex file. Now I get a ris citation with a simple call to

M-x ris-citation-to-bib

Then I enter the url and the function appends the citation to my bibtex file.²

Feel free to integrate it into your own emacs setup (additionally to the GPLv3 you can use any license used by emacswiki or worg).

(defun ris-citation-to-bib (&optional ris-url) 
  "get a ris citation as bibtex in one step. Just call M-x
ris-citation-to-bib and enter the ris url. 
Requires bibutils: http://sourceforge.net/p/bibutils/home/Bibutils/ 
"
  (interactive "Mris-url: ")
  (save-excursion
    (let ((bib-file "/home/arne/aufschriebe/ref.bib")
          (bib-buffer (get-buffer "ref.bib"))
          (ris-buffer (url-retrieve-synchronously ris-url)))
      ; firstoff check if we have the bib buffer. If yes, move point to the last line.
      (if (not (member bib-buffer (buffer-list)))
          (setq bib-buffer (find-file-noselect bib-file)))
      (progn 
        (set-buffer bib-buffer)
        (goto-char (point-max)))
      (if ris-buffer
          (set-buffer ris-buffer))
      (shell-command-on-region (point-min) (point-max) "ris2xml | xml2bib" ris-buffer)
      (let ((pmin (- (search-forward "@") 1))
            (pmax (search-forward "}

")))
        (if (member bib-buffer (buffer-list))
            (progn 
              (append-to-buffer bib-buffer pmin pmax)
              (kill-buffer ris-buffer)
              (set-buffer bib-buffer)
              (save-buffer)
          ))))))

Happy Hacking!

El Kanban Org: parse org-mode todo-states to use org-tables as Kanban tables

Kanban for emacs org-mode.

Update (2013-04-13): Kanban.el now lives in its own repository: on bitbucket and on a statically served http-repo (to be independent from unfree software).

Update (2013-04-10): Thanks to Han Duply, kanban links now work for entries from other files. And I uploaded kanban.el on marmalade.

Some time ago I learned about kanban, and the obvious next step was: “I want to have a kanban board from org-mode”. I searched for it, but did not find any. Not wanting to give up on the idea, I implemented my own :)

The result are two functions: kanban-todo and kanban-zero.

“Screenshot” :)

TODO	DOING	DONE
Refactor in such a way that the
	let Presentation manage dumb sprites
return all actions on every command:
Make the UiState adhere the list of
Turn the model into a pure state

kanban-todo

kanban-todo provides your TODO items as kanban-fields. You can move them in the table without having duplicates, so all the state maintenance is done in the kanban table. Once you are finished, you mark them as done and delete them from the table.

To set it up, put kanban.el somewhere in your load path and (require 'kanban) (more recent but potentially unstable version). Then just add a table like the following:

|   |   |   |
|---+---+---|
|   |   |   |
|   |   |   |
|   |   |   |
|   |   |   |
#+TBLFM: $1='(kanban-todo @# @2$2..@>$>)::@1='(kanban-headers $#)

Click C-c C-c with the point on the TBLFMT line to update the table.

The important line is the #+TBLFM. That says “use my TODO items in the TODO column, except if they are in another column” and “add kanban headers for my TODO states”

The kanban-todo function takes an optional parameter match, which you can use to restrict the kanban table to given tags. The syntax is the same as for org-mode matchers. The third argument allows you to provide a scope, for example a list of files.

To only set the scope, use nil for the matcher.

See C-h f org-map-entries and C-h v org-agenda-files for details.

kanban-zero

kanban-zero is a zero-state Kanban: All state is managed in org-mode and the table only displays the kanban items.

To set it up, put kanban.el somwhere in your load path and (require 'kanban). Then just add a table like the following:

|   |   |   |
|---+---+---|
|   |   |   |
|   |   |   |
|   |   |   |
|   |   |   |
#+TBLFM: @2$1..@>$>='(kanban-zero @# $#)::@1='(kanban-headers $#)

The important line is the #+TBLFM. That says “show my org items in the appropriate column” and “add kanban headers for my TODO states”.

Click C-c C-c with the point on the TBLFMT line to update the table.

The kanban-zero function takes an optional parameter match, which you can use to restrict the kanban table to given tags. The syntax is the same as for org-mode matchers. The third argument allows you to provide a scope, for example a list of files.

To only set the scope, use nil for the matcher.

An example for matcher and scope would be:

#+TBLFM: @2$1..@>$>='(kanban-zero @# $# "1w6" '("/home/arne/.emacs.d/private/org/emacs-plan.org"))::@1='(kanban-headers $#)

See C-h f org-map-entries and C-h v org-agenda-files for details.

Contribute

To contribute to kanban.el, just change the file and write a comment about your changes. Maybe I’ll setup a repo on Bitbucket at some point…

Example

In the Hexbattle game-draft, I use kanban to track my progress:

Table of Contents

• 1 Kanban
• 2 refactor Hexbattle
  • 2.1 Structure plan:
  • 2.2 First all logic from UiState into the model. Model gets all state which is independent of the UI. Any necessary interaction in UiState calls model methods. √
  • 2.3 Refactor in such a way that the UiState and (even more importantly) the input do not need to call the model directly anymore.
    • 2.3.1 let Presentation manage dumb sprites for all chars which are active. Then I can remove a lot of UiState dependent functionality from the chars.
    • 2.3.2 return all actions on every command: command(…) → commands (instance of Commands).
  • 2.4 Make the UiState adhere the list of possible commands.
  • 2.5 Turn the model into a pure state machine.

1 Kanban

❢	STARTED	✔
Refactor in such a way that the
	let Presentation manage dumb sprites
return all actions on every command:
Make the UiState adhere the list of
Turn the model into a pure state

2 refactor Hexbattle    1w6

… and so on …

Advanced usage

“Graphical” TODO states

To make the todo states easier to grok directly you can use unicode symbols for them. Example:

#+SEQ_TODO: ❢ ☯ ⧖ | ☺ ✔ DEFERRED ✘

| ❢ | ☯ | ⧖ | ☺ |
|---+---+---+---|
|   |   |   |   |
#+TBLFM: @1='(kanban-headers $#)::@2$1..@>$>='(kanban-zero @# $#)

In my setup they are ❢ (todo) ☯ (doing) ⧖ (waiting) and ☺ (to report). Not shown in the kanban Table are ✔ (finished), ✘ (dropped) and deferred (later), because they don’t require any action from me, so I don’t need to see them all the time.

Collecting kanban entries via SSH

If you want to create a shared kanban table, you can use the excellent transparent network access options from Emacs tramp to collect kanban entries directly via SSH.

To use that, simply pass an explicit list of files to kanban-zero as 4th argument (if you don’t use tag matching just use nil as 3rd argument). "/ssh:host:path/to/file.org" retrieves the file ~/path/to/file.org from the host.

| ❢ | ☯ |
|---+---|
|   |   |
#+TBLFM: @1='(kanban-headers $#)::@2$1..@>$>='(kanban-zero @# $# nil (list (buffer-file-name) "/ssh:localhost:plan.org"))

Caveeat: all included kanban files have to use at least some of the same todo states: kanban.el only retrieves TODO states which are used in the current buffer.

emacs wanderlust.el setup for reading kmail maildir

This is my wanderlust.el file to read kmail maildirs. You need to define every folder you want to read.

;; mode:--emacs-lisp--
;; wanderlust 
(setq 
  elmo-maildir-folder-path "~/.kde/share/apps/kmail/mail"
          ;; where i store my mail


wl-stay-folder-window t                       ;; show the folder pane (left)
  wl-folder-window-width 25                     ;; toggle on/off with 'i'

wl-smtp-posting-server "smtp.web.de"            ;; put the smtp server here
  wl-local-domain "draketo.de"          ;; put something here...
  wl-message-id-domain "web.de"     ;; ...

  wl-from "Arne Babenhauserheide arne_bab@web.de"                  ;; my From:


;; note: all below are dirs (Maildirs) under elmo-maildir-folder-path 
  ;; the '.'-prefix is for marking them as maildirs
  wl-fcc ".sent-mail"                       ;; sent msgs go to the "sent"-folder
  wl-fcc-force-as-read t               ;; mark sent messages as read 
  wl-default-folder ".inbox"           ;; my main inbox 
  wl-draft-folder ".drafts"            ;; store drafts in 'postponed'
  wl-trash-folder ".trash"             ;; put trash in 'trash'
  wl-spam-folder ".gruppiert/Spam"              ;; ...spam as well
  wl-queue-folder ".queue"             ;; we don't use this

;; check this folder periodically, and update modeline
  wl-biff-check-folder-list '(".todo") ;; check every 180 seconds
                                       ;; (default: wl-biff-check-interval)

;; hide many fields from message buffers
  wl-message-ignored-field-list '("^.*:")
  wl-message-visible-field-list
  '("^\(To\|Cc\):"
    "^Subject:"
    "^\(From\|Reply-To\):"
    "^Organization:"
    "^Message-Id:"
    "^\(Posted\|Date\):"
    )
  wl-message-sort-field-list
  '("^From"
    "^Organization:"
    "^X-Attribution:"
     "^Subject"
     "^Date"
     "^To"
     "^Cc"))

; Encryption via GnuPG

(require 'mailcrypt)
 (load-library "mailcrypt") ; provides "mc-setversion"
(mc-setversion "gpg")    ; for PGP 2.6 (default); also "5.0" and "gpg"

(autoload 'mc-install-write-mode "mailcrypt" nil t)
(autoload 'mc-install-read-mode "mailcrypt" nil t)
(add-hook 'mail-mode-hook 'mc-install-write-mode)

(add-hook 'wl-summary-mode-hook 'mc-install-read-mode)
(add-hook 'wl-mail-setup-hook 'mc-install-write-mode)

;(setq mc-pgp-keydir "~/.gnupg")
;(setq mc-pgp-path "gpg")
(setq mc-encrypt-for-me t)
(setq mc-pgp-user-id "FE96C404")

(defun mc-wl-verify-signature ()
  (interactive)
  (save-window-excursion
    (wl-summary-jump-to-current-message)
    (mc-verify)))

(defun mc-wl-decrypt-message ()
  (interactive)
  (save-window-excursion
    (wl-summary-jump-to-current-message)
    (let ((inhibit-read-only t))
      (mc-decrypt))))

(eval-after-load "mailcrypt"
  '(setq mc-modes-alist
       (append
        (quote
         ((wl-draft-mode (encrypt . mc-encrypt-message)
            (sign . mc-sign-message))
          (wl-summary-mode (decrypt . mc-wl-decrypt-message)
            (verify . mc-wl-verify-signature))))
        mc-modes-alist)))

; flowed text

;; Reading f=f
 (autoload 'fill-flowed "flow-fill")
 (add-hook 'mime-display-text/plain-hook
          (lambda ()
            (when (string= "flowed"
                           (cdr (assoc "format"
                                       (mime-content-type-parameters
                                        (mime-entity-content-type entity)))))
              (fill-flowed))))
; writing f=f
;(mime-edit-insert-tag "text" "plain" "; format=flowed")

(provide 'private-wanderlust)

UPDATE (2012-05-07): ~/.folders

I now use a ~/.folders file, to manage my non-kmail maildir subscriptions, too. It looks like this:

.sent-mail
.~/.local/share/mail/mgl_spam   "mgl spam" 
.~/.local/share/mail/to.arne_bab    "to arne_bab"
.inbox  "inbox" 
.trash  "Trash"
..gruppiert.directory/.inbox.directory/Freunde  "Freunde"
.drafts "Drafts"
..gruppiert.directory/.alt.directory/Posteingang-2011-09-18 "2011-09-18"
.outbox

The mail in ~/.local/share/mail is fetched via fetchmail and procmail to have a really reliable mail fetching system which does not rely on a non-broken database or free space on the disk to keep working…

Insert a scaled screenshot in emacs org-mode

@marjoleink asked on identi.ca¹, if it is possible to use emacs org-mode for showing scaled screenshots inline while writing. Since I thought I’d enjoy some hacking, I decided to take the challenge.

It does not do auto-scaling of embedded images, as far as I know, but the use case of screenshots can be done with a simple function:

(defun org-insert-scaled-screenshot ()
  "Insert a scaled screenshot 
for inline display 
into your org-mode buffer."
  (interactive)
  (let ((filename 
         (concat "screenshot-" 
                 (substring 
                  (shell-command-to-string 
                   "date +%Y%m%d%H%M%S")
                  0 -1 )
                 ".png")))
    (let ((scaledname 
           (concat filename "-width300.png")))

      (shell-command 
       (concat "import -window root " 
               filename))
      (shell-command 
       (concat "convert -adaptive-resize 300 " 
               filename " " scaledname))
      (insert (concat "[[./" scaledname "]]")))))

Now just call M-x org-redisplay-inline-images to see the screenshot (or add it to the function).

In action:

Have fun with Emacs - and happy hacking!

Minimal example for literate programming with noweb in emacs org-mode

If you want to use the literate programming features in emacs org-mode, you can try this minimal example to get started: Activate org-babel-tangle, then put this into the file noweb-test.org:

Minimal example for noweb in org-mode

* Assign 

First we assign abc:

#+begin_src python :noweb-ref assign_abc
abc = "abc"
#+end_src

* Use

Then we use it in a function:

#+begin_src python :noweb tangle :tangle noweb-test.py
def x():
  <<assign_abc>>
  return abc

print(x())
#+end_src

noweb-test.org

Hit C-c C-c to evaluate the source block. Hit C-c C-v C-t to put the expanded code into the file noweb-test.py.

The exported code looks like this:

  def x():
    abc = "abc"
    return abc

  print(x())

noweb-test.py

(html generated with org-export-as-html-to-buffer and slightly reniced to escape the additional parsing I have on my site)

And with org-export-as-pdf we get this:

noweb-test.pdf

Add :results output to the #+begin_src line of the second block to see the print results under that block when you hit C-c C-c in the block.

You can also use properties of headlines for giving the noweb-ref. Org-mode can then even concatenate several source blocks into one noweb reference. Just hit C-c C-x p to set a property (or use M-x org-set-property), then set noweb-ref to the name you want to use to embed all blocks under this heading together.

Note: org-babel prefixes each line of an included code-block with the prefix used for the reference (here <<assign_abc>>). This way you can easily include blocks inside python functions.

Have fun with Emacs and org-mode!

Read your python module documentation from emacs

I just found the excellent pydoc-info mode for emacs from Jon Waltman. It allows me to hit C-h S in a python file and enter a module name to see the documentation right away. If the point is on a symbol (=module or class or function), I can just hit enter to see its docs.

In its default configuration (see the Readme) it “only” reads the python documentation. This alone is really cool when writing new python code, but it s not enough, since I often use third party modules.

And now comes the treat: If those modules use sphinx for documentation (≥1.1), I can integrate them just like the standard python documentation!

It took me some time to get it right, but now I have all the documentation for the inverse modelling framework I contribute to directly at my fingertips: Just hit C-h S ENTER when I’m on some symbol and a window shows me the docs:

The text in this image is from Wouter Peters. Used here as short citation which should be legal almost everywhere under citation rules.

I want to save you the work of figuring out how to do that yourself, so here’s a short guide for integrating the documentation for your python program into emacs.

Integrating your own documentation into emacs

The prerequisite for integrating your own documentation is to use sphinx for documenting your code. See their tutorial for info how to set it up. As soon as sphinx works for you, follow this guide to integrate your docs in your emacs.

Install pydoc-info

First get pydoc-info and the python infofile (adapt this to your local setup):

# get the mode
cd ~/.emacs.d/libs
hg clone https://bitbucket.org/jonwaltman/pydoc-info
# and the pregenerated info-file for python
wget https://bitbucket.org/jonwaltman/pydoc-info/downloads/python.info.gz
gunzip python.info
sudo cp python.info /usr/share/info
sudo install-info --info-dir=/usr/share/info python.info

(I also added pydoc-info as subrepo to my .emacs.d repo to make it easy to transfer my adaption between my different computers)

To build the info file for python yourself, have a look at the Readme.

Turn your documentation into info

Now turn your own documentation into an info document and install it.

Sphinx uses a core configuration file named conf.py. Add the following to that file, replacing all values but index and False by the appropriate names for you project:

# One entry per manual page. 
# list of tuples (startdocname, 
# targetname, title, author, dir_entry, 
# description, category, toctree_only).
texinfo_documents = [
  ('index', # startdocname, keep this!
   'TARGETNAME', # targetname
   u'Long Title', # title
   u'Author Name', # author
   'Name in the Directory Index of Info', # dir_entry
   u'Long Description', # description
   'Software Development', # cathegory
   False), # better keep this, too, i think.
]

Then call sphinx and install the info files like this (maybe adapted to your local setup):

sphinx-build -b texinfo source/ texinfo/ 
cd texinfo
sudo install-info --info-dir=/usr/share/info TARGETNAME.info
sudo cp TARGETNAME.info /usr/share/info/

Activate pydoc-info, including your documentation

Finally add the following to your .emacs (or wherever you store your personal adaptions):

; Show python-documentation as info-pages via C-h S
(setq load-path (cons "~/.emacs.d/libs/pydoc-info" load-path))
(require 'pydoc-info)
(info-lookup-add-help
   :mode 'python-mode
   :parse-rule 'pydoc-info-python-symbol-at-point
   :doc-spec
   '(("(python)Index" pydoc-info-lookup-transform-entry)
     ("(TARGETNAME)Index" pydoc-info-lookup-transform-entry)))

Recipes for presentations with beamer latex using emacs org-mode

I wrote some recipes for creating the kinds of slides I need with emacs org-mode export to beamer latex.

Update: Read ox-beamer to see how to adapt this to work with the new export engine in org-mode 0.8.

The recipes as PDF (21 slides, 247 KiB)

The org-mode sources (12.2 KiB)

Below is an html export of the org-mode file. Naturally it does not look as impressive as the real slides, but it captures all the sources, so I think it has some value.

Note: To be able to use the simple block-creation commands, you need to add #+startup: beamer to the header of your file or explicitely activate org-beamer with M-x org-beamer-mode.

PS: I hereby allow use of these slides under any of the licenses used by worg and/or the emacs wiki.

Hello World

#+LaTeX_CLASS: beamer
#+BEAMER_FRAME_LEVEL: 2

* Hello
** Hello GNU
Nice to see you!

Recipes
#+LaTeX_CLASS: beamer
#+BEAMER_FRAME_LEVEL: 2
* Introduction
** org-mode + beamer =  love
*** Code :BMCOL:
    :PROPERTIES:
    :BEAMER_col: 0.7
    :END:
<example block>
*** Simple block  :BMCOL:B_block:
    :PROPERTIES:
    :BEAMER_col: 0.3
    :BEAMER_env: block
    :END:
it's that easy!

** Two columns - in commands
*** Commands
C-c C-b | 0.7
C-c C-b b
C-n
<eTAB (write example) C-n C-n
*** Result
C-c C-b | 0.3
C-c C-b b
even easier - and faster!

*** Column 1 :B_ignoreheading:BMCOL:
    :PROPERTIES:
    :BEAMER_env: ignoreheading
    :BEAMER_col: 0.5
    :END:

*** One
*** Three                                                           

*** Column 2 :BMCOL:B_ignoreheading:
    :PROPERTIES:
    :BEAMER_col: 0.5
    :BEAMER_env: ignoreheading
    :END:

*** Two
*** Four

*** 
C-c C-b | 0.5 # column
C-c C-b i # ignore heading
*** One 
C-c C-b b # block
*** Three 
C-c C-b b
*** 
C-c C-b | 0.5
C-c C-b i
*** Two 
C-c C-b b
*** Four 
C-c C-b b

*** Code                                                      :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.5
    :BEAMER_envargs: C[t]
    :END:

*** Result                                                    :B_block:BMCOL:
    :PROPERTIES:
    :BEAMER_env: block
    :BEAMER_col: 0.5
    :END:
pretty nice!

#+begin_quote
Emacs org-mode is a 
great presentation tool - 
Fast to beautiful slides.
- Arne Babenhauserheide
#+end_quote

\( 1 + 2 = 3 \) is clear

\[ 1 + 2 \cdot 3 = 7 \]

\( \LaTeX \) gives a space 
after math mode.

\LaTeX{} does it, too.

\LaTeX does not.

At the end of a sentence 
both work.
Try \LaTeX. Or try \LaTeX{}.

Only \( \LaTeX \) and \( \LaTeX{} \) 
also work with HTML export.

: #+bla: foo
: * Example Header

#+begin_example
content
#+end_example

#+bla: foo
* Example Header

content

<Title>

#+startup: beamer
#+LaTeX_CLASS: beamer
#+LaTeX_CLASS_OPTIONS: [bigger]
#+AUTHOR: <empty for none, if missing: inferred>
#+DATE: <empty for none, if missing: today>
#+BEAMER_FRAME_LEVEL: 2
#+TITLE: <causes <Title> to be regular content!>

(org-babel-do-load-languages ; babel, for executing 
 'org-babel-load-languages   ; code in org-mode.
 '((sh . t)
   (emacs-lisp . t)))

(require 'org-latex) ; latex export 
(add-to-list         ; with highlighting
  'org-export-latex-packages-alist '("" "minted"))
(add-to-list 
  'org-export-latex-packages-alist '("" "color"))
(setq org-export-latex-listings 'minted)

(custom-set-variables ; in ~/.emacs, only one instance 
 '(org-export-latex-classes (quote ; in the init file!
    (("beamer" "\\documentclass{beamer}" 
        org-beamer-sectioning))))
 '(org-latex-to-pdf-process (quote 
    ((concat "pdflatex -interaction nonstopmode" 
             "-shell-escape -output-directory %o %f") 
     "bibtex $(basename %b)" 
     (concat "pdflatex -interaction nonstopmode" 
             "-shell-escape -output-directory %o %f")
     (concat "pdflatex -interaction nonstopmode" 
             "-shell-escape -output-directory %o %f")))))

emerge emacs

emerge dev-tex/latex-beamer app-text/texlive

emerge dev-python/pygments

Sending email to many people with Emacs Wanderlust

I recently needed to send an email to many people¹.

Putting all of them into the BCC field did not work (mail rejected by provider) and when I split it into 2 emails, many did not see my mail because it was flagged as potential spam (they were not in the To-Field)².

I did not want to put them all into the To-Field, because that would have spread their email-addresses around, which many would not want³.

So I needed a different solution. Which I found in the extensibility of emacs and wanderlust⁴. It now carries the name wl-draft-send-to-multiple-receivers-from-buffer.

You simply write the email as usual via wl-draft, then put all email addresses you want write to into a buffer and call M-x wl-draft-send-to-multiple-receivers-from-buffer. It asks you about the buffer with email addresses, then shows you all addresses and asks for confirmation.

Then it sends one email after the other, with a randomized wait of 0-10 seconds between messages to avoid flagging as spam.

If you want to use it, just add the following to your .emacs:

(defun wl-draft-clean-mail-address (address)
  (replace-regexp-in-string "," "" address))

(defun wl-draft-send-to-multiple-receivers (addresses)
  (loop for address in addresses do
        (progn 
          (wl-user-agent-insert-header "To" 
            (wl-draft-clean-mail-address address))
          (let ((wl-interactive-send nil))
            (wl-draft-send))
          (sleep-for (random 10)))))

(defun wl-draft-send-to-multiple-receivers-from-buffer 
    (&optional addresses-buffer-name)
  "Send a mail to multiple recipients - one recipient at a time"
  (interactive "BBuffer with one address per line")
  (let ((addresses nil))
    (with-current-buffer addresses-buffer-name
      (setq addresses (split-string (buffer-string) "\n")))
    (if (y-or-n-p (concat "Send this mail to " 
            (mapconcat 'identity addresses ", ")))
        (wl-draft-send-to-multiple-receivers addresses))))

Happy Hacking!

Simple Emacs DarkRoom

I just realized that I let myself be distracted by all kinds of not-so-useful stuff instead of finally getting to type the text I already wanted to transcribe from stenografic at the beginning of … last week.

Screenshot!

Let’s take a break for a screenshot of the final version, because that’s what we want from any program :)

As you can see, the distractions are removed — the screenshot is completely full screen and only the text is left. If you switch to the minibuffer (i.e. via M-x), the status bar (modeline) is shown.

Background

To remove the distractions I looked again at WriteRoom and DarkRoom and similar which show just the text I want to write. More exactly: I thought about looking at them again, but at second thought I decided to see if I could not just customize emacs to do the same, backed with all the power you get from several decades of being THE editor for many great hackers.

It took some googling and reading emacs wiki, and then some Lisp-hacking, but finally it’s 4 o’clock in the morning and I’m writing this in my own darkroom mode¹, toggled on and off by just hitting F11.

Implementation

I build on hide-mode-line (livejournal post or webonastick) as well as the full-screen info in the emacs wiki.

The whole code just takes 29 lines of code plus 10 lines comments and whitespace:

; hide mode line, from 
; http://dse.livejournal.com/66834.html 
; http://webonastick.com
(autoload 'hide-mode-line "hide-mode-line" nil t)

; fullscreen, taken from
; http://www.emacswiki.org/emacs/FullScreen#toc1
(defun toggle-fullscreen (&optional f)
  (interactive)
  (let ((current-value (frame-parameter nil 'fullscreen)))
    (set-frame-parameter nil 'fullscreen
             (if (equal 'fullboth current-value)
                 (if (boundp 'old-fullscreen) old-fullscreen nil)
               (progn (setq old-fullscreen current-value)
                  'fullboth)))))

; simple darkroom with fullscreen, 
; fringe, mode-line, menu-bar and scroll-bar hiding.
(defvar darkroom-enabled nil)

(defun toggle-darkroom ()
  (interactive)
  (if (not darkroom-enabled)
      (setq darkroom-enabled t)
    (setq darkroom-enabled nil))
  (toggle-fullscreen)
  (hide-mode-line)
  (if darkroom-enabled
      (progn
        (fringe-mode 'both)
        (menu-bar-mode -1)
        (scroll-bar-mode -1)
        (set-fringe-mode 200))
    (progn 
      (fringe-mode 'default)
      (menu-bar-mode)
      (scroll-bar-mode t)
      (set-fringe-mode 8))))

; Activate with F11 - enhanced fullscreen :)
(global-set-key [f11] 'toggle-darkroom)

Also I now activated cua-mode to make it easier to interact with other programs: C-c and C-x now copy/cut when the mark is active. Otherwise they are the usual prefix keys. To force them to be the prefix keys, I can use control-shift-c/-x. I thought this would disturb me, but it does not.

To make it faster, I also told cua-mode to have a maximum delay of 50ms, so I don’t feel the delay. Essentially I just put this in my ~/.emacs:

(cua-mode t)
(setq cua-prefix-override-inhibit-delay 0.005)

Epilog

Well, did this get me to transcribe the text? Not really, since I spent the time building my own DarkRoom/WriteRoom, but I enjoyed the little hacking and it might help me get it done tomorrow - and get far more other stuff done.

And it is really fun to write in DarkRoom mode ;)

PS: If you like the simple darkroom, please leave a comment!

I hereby declare that anyone is allowed to use this post and the screenshot under the same licensing as if it had been written in emacswiki.

Wish: KDE with Emacs-style keyboard shortcuts

I would love to be able to use KDE with emacs-style keyboard shortcuts, because Emacs offers a huge set of already clearly defined shortcuts for many different situations. Since its users tend to do very much with the keyboard alone, even more obscure tasks are available via shortcuts.

I think that this would be useful, because Emacs is like a kind of nongraphical desktop environment itself (just look at emacspeak!). For all those who use Emacs in a KDE environment, it could be a nice timesaver to be able to just use their accustomed bindings.

It also has a mostly clean structure for the bindings:

• "C-x anything" does changes which affect things outside the content of the current buffer.
• "C-anything but x" acts on the content you're currently editing.
• "C-c anything" is kept for specific actions of programs. For example "C-c C-c" in an email sends the email, while "C-c C-c" in a version tracking commit message finishes the message and starts the actual commit.
• "M-x" opens a 'command-selection-dialog' (just like alt-f2).
• "M-anything but x" is a different flavor of "C-anything but x". For example "C-f" moves the cursor one character forward, while "M-f" moves one word forward. "C-v" moves one page forward, while "M-v" moves one page backwards.

On the backend side, this would require being able to define multistep shortcuts. Everything else is just porting the emacs shortcuts to KDE actions.

The actual porting of shortcuts would then require mapping of the Emacs commands to KDE actions.

Some examples:

• "C-s" searches in a file. Replaces C-f.
• "C-r" searches backwards.
• "C-x C-s" saves a file -> close. Replaces C-w.
• "C-x C-f" opens a file -> Open. Replaces C-o.
• "C-x C-c" closes the program -> quit. Replaces C-q.
• "C-x C-b" switches between buffers/files/tabs -> switch the open file. Replaces alt-right_arrow and a few other (to my knowledge) inconsistent bindings.
• "C-x C-2" splits a window (or part of a window) vertically. "C-x C-o" switches between the parts. "C-x C-1" undoes the split and keeps the currently selected part. "C-x C-0" undoes the split and hides the currently selected part.

50€ for the Freenet Project - and against censorship

As I pledged¹, I just donated to freenet 50€ of the money I got back because I cannot go to FilkCONtinental. Thanks go to Nemesis, a proud member of the “FiB: Filkers in Black” who will take my place at the Freusburg and fill these old walls with songs of stars and dreams - and happy laughter.

It’s a hard battle against censorship, and as I now had some money at hand, I decided to do my part (freenetproject.org/donate.html).

A vision for a social Freenet with WoT, FreeTalk and Sone

I let my thought wander a bit around the question how a social Freenet (2.0 ;) ) could look from the view of a newcomer.

I imagine myself installing freenet. The first thing to come up after starting it is the node page. (italic Text in brackets is a comment. The links need a Freenet running on 127.0.0.1 to work)

“Welcome to Freenet, where no one can tell you’re reading”

“Freenet tries hard to project your privacy. Therefore we created a pseudonymous ID for you. Its name is Gandi Schmidt. Visit the [your IDs site] to see a legend we prepared for you. You can use this legend as fictional background for your ID, if you are really serious about staying anonymous.”

(The name should be generated randomly for each ID. A starting point for that could be a list of scientists from around the world compiled from the wikipedia (link needs freenet). The same should be true for the legend, though it is harder to generate. The basic information should be a quote (people remember that), a job and sex, the country the ID comes from (maybe correlated with the name) and a hobby.)

“During the next few restarts, Freenet will ask you to solve various captchas to prove that you are indeed human. Once enough other nodes successfully confirmed that you are human, you will be granted access to the forums and microblogging. This might take a few hours to a few days.”

(as soon as the ID has sufficient trust, automatically activate the FreeTalk and Sone plugins)

“Note that other nodes don’t know who you are. They don’t know your IP, nor your real identity. The only thing they know is that you exist, that you can solve captchas and how to send you a message.”

“You can create additional IDs at any time and give them any name and legend you choose by adding it on the WebOfTrust-page. Each new ID has to verify for itself that it’s human, though. If you carefully keep them seperate, others can only find out with a lot of effort that your IDs are related. Mind your writing style. In doubt, keep your sentences short. To make it easier for you to stay anonymous, you can autogenerate Name and Legend at random. Don’t use the nicest from many random trials, else you can be traced by the kind of random IDs you select.”

“While your humanity is being confirmed, you can find a wealth of content on the following indexes, some published anonymously, some not. If you want to publish your own anonymous site, see Upload a Freesite. The list of indexes uses dynamic bookmarks. You get notified whenever a bookmarked site (like the indexes below) gets updated.”

“Note: If you download content from freenet, it is being cached by other nodes. Therefore popular content is faster than rare content and you cannot overload nodes by requesting their data over and over again.”

“You are currently using medium security in the range from low to high.”

“In this security level, seperated IDs are no perfect protection of your anonymity, though, since other members might not be able to see what you do in Freenet, but they can know that you use freenet in the first place, and corporations or governments with medium sized infrastructure can launch attacks which might make it possible to trace your contributions and accesses. If you want to disappear completely from the normal web and keep your freenet usage hidden, as well as make it very hard to trace your contributions, to be able to really exercise your right of free speech without fearing repercussions, you can use Freenet as Darknet — the more secure but less newcomer friendly way to use freenet; the current mode is Opennet.”

“To enter the Darknet, you add people you know and trust personally as your darknet friends. As soon as you have enough trusted friends, you can increase the security level to high and freenet will only connect to your trusted friends, making you disappear from the regular internet. The only way to tell that you are using freenet will then be to force your ISP to monitor all traffic coming from your computer.”

“And once transport plugins are integrated, steganography will come into reach and allow masking your traffic as regular internet usage, making it very hard to distinguish freenet from encrypted internet-telephony. If you want to help making this a reality in the near future, please consider contributing or donating to freenet.”

“Welcome to the pseudonymous web where no one can know who you are, but only that you are always using the same ID — if you do so.”

“To show this welcome message again, you can at any time click on Intro in the links.”

What do you think? Would this be a nice way to integrate WoT, FreeTalk, Sone and general user education in a welcome message, while adding more incentive to keep the node running?

PS: Also posted in Freetalk and in Sone – the links need a running Freenet to work.

PPS: This vision is not yet a reality, but all the necessary infrastructure is already in place and working in Freenet. You can already do everything described in here, just without the nice guide and the level of integration (for example activating plugins once you have proven your humanity, which equals enough trust by others to be actually seen).

Anonymous code collaboration with Mercurial and Freenet

There is now a new Mercurial extension called "infocalypse" (which should keep working after the information apocalypse).

It offers "fn-push" and "fn-pull" as an optimized way to store code in freenet: bundles are inserted and pulled one after the other. An index tells infocalypse in which order to pull the bundles. It makes using Mercurial in freenet far more efficient and convenient.

Also you can use it to publish collaborative anonymous websites like the freefaq and Technophob.

And it is a perfect fit for the workflow automatic trusted group of committers.

Otherwise it offers the same features as FreenetHG.

Using FreenetHG you can collaborate anonymously without having to give everyone direct write access to your code.

To work with others, you simply setup a local repository for your own work and use FreenetHG to upload your code automatically into Freenet under your private ID. Others can then access your code with the corresponding public ID, do their changes locally and publish them in their own anonymous repository.

You then pull changes you like into your repository and publish them again under your key.

FreenetHG uses freenet which offers the concept of pseudonymity to make anonymous communication more secure and Mercurial to allow for efficient distributed collaboration.

With pseudonymity you can't find out whom you're talking to, but you know that it is the same person, and with distibuted collaboration you don't need to let people write to your code directly, since every code repository is a full clone of the main repository.

Even if the main repository should go down, every contributor can still work completely unhindered, and if someone else breaks things in his repository, you can simply decide not to pull the changes from him.

What you need

To use FreenetHG you obviously need a running freenet node and a local Mercurial installation. Also you need the FreenetHG plugin for Mercurial and PyFCP which provides Python bindings for Freenet.

• get FreenetHG (the link needs a running freenet node on 127.0.0.1)

• alternatively just do

  hg clone static-http://127.0.0.1:8888/USK@fQGiK~CfI8zO4cuNyhPRLqYZ5TyGUme8lMiRnS9TCaU,E3S1MLoeeeEM45fDLdVV~n8PCr9pt6GMq0tuH4dRP7c,AQACAAE/freenethg/1/

Setup a simple anonymous workflow

To guide you through the steps, let's assume we want to create the anonymous repository "AnoFoo".

After you got all dependencies, you need to activate the FreenetHG plugin in your ~/.hgrc file

[extensions]
freenethg = path/to/FreenetHG.py

You can get the FreenetHG.py from the freenethg website or from the Mercurial repository you cloned.

Now you setup your anofoo Mercurial repository:

hg init AnoFoo

As a next step we create some sections in the .hg/hgrc file in the repository:

[ui]

[freenethg]

[hooks]

Now we enter the repository and use the setup wizard

cd AnoFoo
hg fcp-setupwitz

The setup wizard asks us for your username to use for this repository (to avoid accidently breaking our anonymity), the address to our freenet instance and for the path to our repository on freenet.

The default answers should fit. The only one where we have to set something else is the project name. There we enter AnoFoo.

Since we don't yet have a freenet URI for the repository, we just answer '.' to let FreenetHG generate one for us. That's also the default answer.

The commit hook makes sure that we don't commit with another but the selected username.

Also the wizard will print a line like the following:

Request uri is: USK@xlZb9yJbGaKO1onzwawDvt5aWXd9tLZRoSoE17cjXoE,zFqFxAk15H-NvVnxo69oEDFNyU9uNViyNN5ANtgJdbU,AQACAAE/freenethg_test/1/

This is the line others can use to clone your project and pull from it.

And with this we finished setting up our anonymous collaboration repository.

When we commit, every commit will directly be uploaded into Freenet.

So now we can pass the freenet Request uri to others who can clone our repository and setup their own repositories in freenet. When they add something interesting, we then pull the data from their Request uri and merge their code with ours.

Setup a more convenient anonymous workflow

This workflow is already useful, but it's a bit inconvenient to have to wait after each commit until your changes have been uploaded. So we'll now change this basic workflow a bit to be able to work more conveniently.

First step: clone our repositories to a backup location:

hg clone AnoFoo BackFoo

Second step: change our .hg/hgrc to only update when we push to the backup repository, and add the default-push path to the backup repository:

[paths]
default-push = ../BackFoo

[hooks]                                                               
pretxncommit = python:freenethg.username_checker                      
outgoing = python:freenethg.updatestatic_hook                           

[ui]
username = anonymuse

[freenethg]
commitusername = anonymuse
inserturi = USK@VERY_LONG_PRIVATE_KEY/AnoFoo/1/

Changes: We now have a default-push path, and we changed the "commit" hook to an "outgoing" hook which is evoked everytime changes leave this repository. It will also be evoked when someone pulls from this repo, but not when we clone it locally.

Now our commits roll as fast as we're used to from other Mercurial repositories and freenethg will make sure we don't use the wrong username.

When we want to anonymously publish the repository we then simply use

hg push

This will push the changes to the backup and then upload it to your anonymous repository.

And now we finished setting up our reopsitory and can begin using an anonymous and almost infinitely scaleable workflow which only requires our freenet installation to be running when we push the code online.

One last touch: If an upload should chance to fail, you can always repeat it manually with

hg fcp-uploadstatic

Time to go

...out there and do some anonymous coding (Maybe with the workflow automatic trusted group of committers).

Happy hacking!

And if this post caught your interest or you want to say anything else about it, please write a comment.

Also please have a look at and vote for the wish to add a way to contribute anonymously to freenet, to make it secure against attacks on developers.

And last but not least: vote for this article on digg and on yigg.

Background of Freenet Routing and the probes project (GSoC 2012)

The probes project is a google summer of code project of Steve Dougherty intended to optimize the network structure of freenet. Here I will give the background of his project very briefly:

The Small World Structure

Freenet organizes nodes by giving them locations - like coordinates. The nodes know some others and can send data only to those, to which they are connected directly. If your node wants to contact someone it does not know directly, it sends a message to one of the nodes it knows and asks that one to forward the message. The decision whom to ask to forward the message is part of the routing.

And the routing algorithm in Freenet assumes a small world network: Your node knows many people who are close to you and a few who are far away. Imagine that as knowing many people in your home town and few in other towns. There is mathematical proof, that the routing is very efficient and scales to billions of users - if it really operates on a small world network.

So each freenet node tries to organize its connections in such a way, that it is connected to many nodes close by and some from far away.⁽¹⁾ The structure of the local connections of your own node can be characterized by the link length distribution: “How many short and how many long connections do you have?”

Probes and their Promise

The probes project from Steve is to analyze the structure of the network and the structure of the local connections of nodes in an anonymous way to improve the self-organization algorithm in freenet. The reason is that if the structure of the network is no small world network, the routing algorithm becomes much less efficient.

That in turn means that if you want to get some data on the network, that data has to travel over far more intermediate nodes, because freenet cannot determine the shortest route. And if the data has to travel over more nodes, it consumes more bandwidth and takes longer to reach you. In the worst case it could happen that freenet does not find the data at all.

To estimate the effect of that, you can look at the bar chart The Seeker linked to:

Low is an ideal structure with 16 connections per node, Conforming is the measured structure with about 17 connections per node (a cluster with 12, one with ~25). Ideally we would want Normal with 26 connections per node and an ideal structure. High is 86 connections. The simulated network sizes are 6000 nodes (Small), 18 000 (Normal, as measured), 36 000 (Large). Fewer hops is better.

It shows how many steps a request has to take to find some content. “Conforming” is the actually measured structure. “low”, “normal” and “high” shows the number of connections per node in an optimal network: 16, 26 and 86. The actually measured mean number of connections in freenet is similar to “low”, so that’s the bar with which we need to compare the “confirming” bar to see the effect of the suboptimal structure. And that effect is staggering: By default a request needs about two times as many steps in the real world than it would need in an optimally structured network.

Practically: If freenet would manage to get closer to the optimal structure, it could double its speed and cut the reaction times by factor 2. Without changing anything else - and also without changing the local bandwidth consumption: You would simply get your content much faster.

If we would manage to increase the mean number of connections to about 26 (that’s what a modern DSL connection can manage without too many ill effects), we could double the speed and half the reaction times again (but that requires more bandwidth in the nodes who currently have a low number of connections: Many have only about 12 connections, many have about 25 or so, few have something in between).

Essentially that means we could gain factor 2 to factor 4 in speed and reaction times. And better scaleability (compare the normal and the large network).

Note ⁽¹⁾: Network Optimization using Only Local Knowledge

To achieve a good local connection-structure, the node can use different strategies for Opennet and Darknet (this section is mostly guessed, take it with a grain of salt. I did not read the corresponding code).

In Opennet it can look if it finds nodes which would improve its local structure. If it finds one, it can replaces the local connection, which distorts its local structure the most, with the new connection.

In Darknet on the other hand, where it can only connect to the folks it already knows, it looks for locations of nodes it hears about. It then checks if its local connection would be better if it had that other nodes location. In that case, it asks the other node if it would agree to swap its location with it (without changing any real connections: It only changes the notion where it lives. As if you would swap the flat with someone else but without changing who your friends are. Afterwards both the other one and you live closer to your respective friends).

In short: In Opennet, Freenet changes to whom it is connected in order to achieve a small world structure: It selects its friends based on where it lives. In Darknet it swaps its location with stranges to live be closer to its friends.

Bootstrapping the Freenet WoT with GnuPG - and GnuPG with Freenet

Intro

When you enter the freenet Web of Trust, you first need to get some trust from people by solving captchas. And even when people trust you somehow, you have no way to prove your identity in an automatic way, so you can’t create identities which freenet can label as trusted without manual intervention from your side.

Proposal

To change this, we can use the Web of Trust used in GnuPG to infer trust relationships between freenet WoT IDs.

Practically that means:

• Write a message: “I am the WoT ID USK@” (replace with the public key of your WoT ID).
• Sign that message with a GnuPG key you want to connect to the ID. The signature proves, that you control the GnuPG key.
• Upload the signed message to your WoT key: USK@/bootstrap/0/gnupg.asc. To make this upload, you need the private key of the ID, so the upload proves, that you control the WoT ID.

Now other people can download the file from you, and when they trust the GnuPG key, they can transfer their trust to the freenet WoT-ID.

Automatic

Ideally all this should be mostly automatic:

• click a link in the freenet interface and select the WoT ID to have freenet create the file and run your local GnuPG program.
• Then select your GnuPG key in the GnuPG program and enter your password.
• Finally check the information to be inserted and press a button to start the upload.

As soon as you have a GnuPG key connected with your WoT ID, freenet should scout all other WoT IDs for gnupg keys and check if the local GnuPG key you assigned to your WoT ID trusts the other key. If yes, give automatic trust (real person → likely no spammer).

Anonymously

To make the connection one-way (bootstrap the WoT from GnuPG, but not expose the key), you might be able to encrypt the message to all people who signed your GnuPG key. Then these can recognize you, but others cannot.

This will lose you the indirect trust in the GnuPG web-of-trust, though.

I hope this bootstrap-WoT draft sounded interesting :)

Happy hacking!

Effortless password protected sharing of files via Freenet

Often you want to exchange some content only with people who know a given password and make it accessible to everyone in your little group but invisible to the outside world.

Until yesterday I thought that problem slightly complex, because everyone in your group needs a given encryption program, and you need a way to share the file without exposing the fact that you are sharing it.

Then I learned two handy facts about Freenet:

• <ArneBab> evanbd: If I insert a tiny file without telling anyone the key, can they get the content in some way?
  <evanbd> ArneBab: No.

• <toad_> dogon: KSK@<any string of text> -> generate an SSK private key from the hash of the text
  <toad_> dogon: if you know the string, you can both insert and retrieve it

In other words: Just inserting a file into freenet using the key KSK@<password> creates an invisible, password protected file which is shared over Freenet.

The file is readable and (within limits¹) writeable by everyone who knows the password, but invisible to everyone else.

To upload a file as KSK, just go to the filesharing tab, click “upload a file”, switch to advanced mode and enter the KSK key.

Or simply click here (requires freenet to be running on your computer with default settings).

It’s strange to think that I only learned this after more than 7 years of using Freenet. How many more nuggets might be hidden there, just waiting for someone to find them and document them in a style which normal users understand?

Freenet is a distributed datastore which can find and transfer data efficiently on restricted routes (search for meshnet scaling to see why that type of routing is really hard), and it uses a WebOfTrust for real-life spam-resistance without the need for a central authority (look at your mailbox to see how hard that is, even with big money).

How many more complex problems might it already have solved as byproduct in the search for censorship resistance?

So, what’s still to be said? Well, if Freenet sounds interesting: Join in!

Exploring the probability of successfully retrieving a file in freenet, given different redundancies and chunk lifetimes

In this text I want to explore the behaviour of the degrading yet redundant anonymous file storage in Freenet. It only applies to files which were not subsequently retrieved.

Every time you retrieve a file, it gets healed which effectively resets its timer as far as these calculations here are concerned. Due to this, popular files can and do live for years in freenet.

# encoding: utf-8
from spielfaehig import spielfähig
from collections import defaultdict
data = []
res = []
for chunknumber in range(5, 105, 5):...
byred = defaultdict(list)
for num, prob, red, retrieval in data:...
csv = "; num prob retrieval"
for red in byred:...

# now plot the files

plotcmd = """
set term png
set width 15
set xlabel "chunk probability"
set ylabel "retrieval probability"
set output freenet-prob-redundancy-2.png
plot "2.csv" using 2:3 select ($1 == 5) title "5 chunks", "" using 2:3 select ($1 == 10) title "10 chunks", "" using 2:3 select ($1 == 30) title "30 chunks", "" using 2:3 select ($1 == 100) title "100 chunks"
set output freenet-prob-redundancy-3.png
plot "3.csv" using 2:3 select ($1 == 5) title "5 chunks", "" using 2:3 select ($1 == 10) title "10 chunks", "" using 2:3 select ($1 == 30) title "30 chunks", "" using 2:3 select ($1 == 100) title "100 chunks"
set output freenet-prob-redundancy-4.png
plot "4.csv" using 2:3 select ($1 == 5) title "5 chunks", "" using 2:3 select ($1 == 10) title "10 chunks", "" using 2:3 select ($1 == 30) title "30 chunks", "" using 2:3 select ($1 == 100) title "100 chunks"
"""
with open("plot.pyx", "w") as f:...

from subprocess import Popen
Popen(["pyxplot", "plot.pyx"])

from spielfaehig import spielfähig
return spielfähig(0.99, 100, 100)
→ 0.366032341273

from spielfaehig import spielfähig
return spielfähig(0.999, 100, 100)
→ 0.904792147114

from spielfaehig import spielfähig
return spielfähig(0.99, 101, 100)
→ 0.732064682546

from spielfaehig import spielfähig
return spielfähig(0.995, 101, 100)
→ 0.908655654736

from spielfaehig import spielfähig
return spielfähig(0.984, 101, 100)
→ 0.518183035909

from spielfaehig import spielfähig
return spielfähig(0.712, 150, 100)
→ 0.904577767501

from spielfaehig import spielfähig
return spielfähig(0.663, 150, 100)
→ 0.500313163333

# encoding: utf-8
plotcmd = """
set term png
set width 15
set xlabel "redundancy"
set ylabel "lifetime [weeks]"
set output "freenet-prob-function.png"
set xrange [0:10]
plot (2/x) * 4 * (2**((0.5-1/x)/0.2))
"""
with open("plot.pyx", "w") as f:...

from subprocess import Popen
Popen(["pyxplot", "plot.pyx"])

Freenet anonymity: Best case and Worst case

As the i2p people say, anynomity is no boolean. Freenet allows you to take it a good deal further than i2p or tor, though. If you do it right.

• Worst case: If all of Apple would want to find you, because you declared that you would post the videos of the new iDing - and already sent them your videos as teaser before starting to upload them from an Apple computer (and that just after they lost their beloved dictator), you might be in problems if you use Opennet. You are about as safe as with tor or i2p.

• Best case: If a local politician would want to find you, after you uploaded proof that he takes bribes, and you compressed these files along with some garbage data and used Freenet in Darknet-mode with connections only to friends who would rather die than let someone take over their computer, there’s no way in hell, you’d get found due to freenet (the file data could betray you, or they could find you by other means, but Freenet won’t be your weak spot).

Naturally real life is somewhere in-between.

Things which improve anonymity a lot in the best case:

• Don’t let others know the data you are going to upload before the upload finished (would allow some attacks).
• Use only Darknet with trusted friends (Darknet means that you connect only to people you know personally. For that it is necessary to know other people who use Freenet).
• Upload small files so the time in which you are actively uploading is short.

Implied are:

• Use an OS without trojans. So no Windows. (Note: Linux can be hacked, too, but it is far less likely to already have been compromised)
• Use no Apple devices. You don’t control them yourself and can’t know what they have under the hood. (You are compromised from the time you buy them)
• If you use Android, flash it yourself to give it an OS you control (Freenet is not yet available for Android. That would be a huge task).
• Know your friends.

Important questions to ask:

• Who would want to find you?
• How much would they invest to find you?
• Do they already try to monitor Freenet? (in that case uploading files with known content would be dangerous)
• Do they already know you personally? If yes and if they might have already compromised your computer or internet connection, you can’t upload anything anonymously anywhere. In that case, never let stuff get onto your computer in the first place. Let someone else upload it, who is not monitored (yet).
• Can they eavesdrop on your internet connection? Then they might guess that you use Freenet from the amount of encrypted communication you do and might want to bug your computer just in case you want to use freenet against them some day.

See the Security Summary (mostly possible attacks) in the freenet wiki for details.

Freenet: WoT, database error, recovery patch

I just had a database error in WoT (the Freenet generic Web of Trust plugin) and couldn’t access one of my identities anymore (plus I didn’t have a backup of its private keys though it told me to keep backups – talk about carelessness :) ).

I asked p0s on IRC and he helped me patch together a WoT which doesn’t access the context for editing the ID (and in turn misses some functionality). This allowed me to regain my IDs private key and with that redownload my ID from freenet.

I didn’t want that patch rotting on my drive, so I uploaded it here: disable-context-checks-regain-keys.path

Applied to revision 4f84492d277e25618003e0e5a0cb14159a50535d of WoT staging.

Essentially it just comments out some stuff.

Infocalypse - Make your code survive the information apocalypse

Anonymous DVCS in the Darknet.

This is a mirror of the documentation of the infocalypse extension for Mercurial written by djk - published here with his permission. It is licensed solely under the GPLv2 or later.

Introduction

The Infocalypse 2.0 hg extension is an extension for Mercurial that allows you to create, publish and maintain incrementally updateable repositories in Freenet.

Your code is then hosted decentrally and anonymously, making it just as censorship-resistant as all other content in Freenet.

It works better than the other DVCS currently available for Freenet.

Most of the information you will find in this document can also be found in the extension's online help. i.e.:

hg help infocalypse

HOWTO: Infocalypse 2.0 hg extension

Table of Contents

• Introduction
• Requirements
• Installation
• Updating
• Background
• Basic Usage
  • Generating a new private key
  • Creating a new repository
  • Pushing to a repository
  • Pulling from a repository
• Using FMS to send and receive update notifications
  • The update trust map
  • Reading other people's notifications
  • Posting your own notifications
  • Pulling an announced repository
• Reinserting and 'sponsoring' repositories
• Forking a repository onto a new USK
• Sharing private keys
• Inserting a freesite
• Risks
• Advocacy
• Source Code
• Fixes and version information
• Freenet-only links
• Contact

Requirements

The extension has the following dependencies:

• Freenet
  You can more information on Freenet here:
  
  http://freenetproject.org/ [HTTP Link!]

• Python
  I test on Python 2.5.4 and 2.6.1. Any 2.5.x or later version should work. Earlier versions may work.

  You probably won't have to worry about installing Python. It's included in the Windows binary Mercurial distributions and most *nix flavor OS's should have a reasonably up to date version of Python installed.

• Mercurial
  You can find more information on Mercurial here:
  
  http://mercurial.selenic.com/ [HTTP Link!]

  Version 1.0.2 won't work.

  I use version 1.2.1 (x86 Gentoo) on a daily basis. Later versions should work.

  I've smoke tested 1.1.2 (on Ubuntu Jaunty Jackalope) and 1.3 (on Widows XP) without finding any problems.

• FMS
  Installation of the Freenet Messaging System (FMS) is optional but
  highly recommended. The hg fn-fmsread and hg fn-fmsnotify commands won't work without FMS. Without fn-fmsread it is extremely difficult to reliably detect repository updates.

  The official FMS freesite is here:

  USK@0npnMrqZNKRCRoGojZV93UNHCMN-6UU3rRSAmP6jNLE,~BG-edFtdCC1cSH4O3BWdeIYa8Sw5DfyrSV-TKdO5ec,AQACAAE/fms/106/
  
  

[TOC]

Installation

You checked the requirements and understandthe risks right?

Here are step-by-step instructions on how to install the extension.

• Download the bootstrap hg bundle:
  

  CHK@S~kAIr~UlpPu7mHNTQV0VlpZk-f~z0a71f7DlyPS0Do,IB-B5Hd7WePtvQuzaUGrVrozN8ibCaZBw3bQr2FvP5Y,AAIC--8/infocalypse2_1723a8de6e7c.hg
      

  You'll get a Potentially Dangerous Content warning from fproxy because the mime type isn't set. Choose 'Click here to force your browser to download the file to disk.'.

  I'll refer to the directory that you saved the bundle file to as DOWNLOAD_DIR.

• Create an empty directory where you want to install the extension.
  I'll refer to that directory as INSTALL_DIR in the
  rest of these instructions.

• Create an empty hg repository there. i.e.:
  

  cd INSTALL_DIR
  hg init
  

• Unbundle the bootstrap bundle into the new repository. i.e:
  

  hg pull DOWNLOAD_DIR/infocalypse2_1723a8de6e7c.hg
  hg update
  

• Edit the '[extensions]' section of your .hgrc/mercurial.ini
  file to point to the infocalypse directory in the unbundled source.

  # .hgrc/mercurial.ini snippet
  [extensions]
  infocalypse = INSTALL_DIR/infocalypse
  

  where INSTALL_DIR is the directory you unbundled into.

  If you don't known where to find/create your .hgrc/mercurial.ini file this link may be useful:
  
  http://www.selenic.com/mercurial/hgrc.5.html [HTTP Link!]

• Run fn-setup to create the config file and temp dir. i.e.
  

  hg fn-setup
     

  If you run your Freenet node on another machine or on a non-standard port you'll need to use the --fcphost and/or --fcpport parameters to set the FCP host and port respectively.

  By default fn-setup will write the configuration file for the extension (.infocalype on *nix, infocalypse.ini on Windows) into your home directory and also create a temp directory called infocalypse_tmp there.

  You can change the location of the temp directory by using the --tmpdir argument.

  If you want to put the config file in a different location set the cfg_file option in the [infocalypse] section of your .hgrc/mercurial.ini file before running fn-setup.

  Example .hgrc entry:
  # Snip, from .hgrc
  [infocalypse]
  cfg_file = /mnt/usbkey/s3kr1t/infocalypse.cfg

• Edit the fms_id and possibly fms_host/fms_port information in the.infocalyse/infocalypse.ini file. i.e.:
  
  

  # Example .infocalypse snippet
  fms_id = YOUR_FMS_ID
  
  fms_host = 127.0.0.1
  fms_port = 1119
  

  where YOUR_FMS_ID is the part of your fms id before the '@' sign.

  If you run FMS with the default settings on the same machine you are running
  Mercurial on you probably won't need to adjust the fcp_host or fcp_port.

  You can skip this step if you're not running fms.

• Read the latest know version of the extension's repository USK index from FMS.
  

  hg fn-fmsread -v
  

  You can skip this step if you're not running fms.

• Pull the latest changes to the extension from Freenet for the first time. Don't skip this step! i.e.:
  

  hg fn-pull --aggressive --debug --uri USK@kRM~jJVREwnN2qnA8R0Vt8HmpfRzBZ0j4rHC2cQ-0hw,2xcoQVdQLyqfTpF2DpkdUIbHFCeL4W~2X1phUYymnhM,AQACAAE/infocalypse.hgext.R1/41
  hg update
  

  You may have trouble finding the top key if you're not using fn-fmsread. Just keep retrying. If you know the index has increased, use the new index in the URI.

  After the first pull, you can update without the URI.

[TOC]

Updating

This extension is under active development. You should periodically update to get the latest bug fixes and new features.

Once you've installed the extension and pulled it for the first time, you can get updates by cd'ing into the initial INSTALL_DIRand typing:

hg fn-fmsread -vhg fn-pull --aggressive hg update

If you're not running FMS you can skip the fn-fmsread step. You may have trouble getting the top key. Just keep retrying.

If you're having trouble updating and you know the index has increased, use the full URI with the new index as above.

[TOC]

Background

Here's background information that's useful when using the extension.See the
Infocalypse 2.0 hg extension page on my freesite for a more detailed description of how the extension works.

Repositories are collections of hg bundle files

An Infocalypse repository is just a collection of hg bundle files which have been inserted into Freenet as CHKs and some metadata describing how to pull the bundles to reconstruct the repository that they represent. When you 'push' to an infocalypse repository a new bundleCHK is inserted with the changes since the last update. When you 'pull', only the CHKs for bundles for changesets not already in the local repository need to be fetched.

Repository USKs

The latest version of the repository's metadata is stored on a Freenet Updateable Subspace Key (USK) as a small binary file.

You'll notice that repository USKs end with a number without a trailing '/'. This is an important distinction. A repository USK is not a freesite. If you try to view one with fproxy you'll just get a 'Potentially Dangerous Content' warning. This is harmless, and ugly but unavoidable at the current time because of limitation in fproxy/FCP.

Repository top key redundancy

Repository USKs that end in *.R1/<number> are inserted redundantly, with a second USK insert done on *.R0/<number>. Top key redundancy makes it easier for other people to fetch your repository.

Inserting to a redundant repository USK makes the inserter more vulnerable to
correlation attacks. Don't use '.R1' USKs if you're worried about this.

Repository Hashes

Repository USKs can be long and cumbersome. A repository hash is the first 12 bytes of the Sha1 hash of the zero index version of a repository USK. e.g.:

SHA1( USK@kRM~jJVREwnN2qnA8R0Vt8HmpfRzBZ0j4rHC2cQ-0hw,2xcoQVdQLyqfTpF2DpkdUIbHFCeL4W~2X1phUYymnhM,AQACAAE/infocalypse.hgext.R1/0 )
  == 'be68e8feccdd'

You can get the repository hash for a repository USK using:

hg fn-info

from a directory the repository USK has been fn-pull'd into.

You can get the hashes of repositories that other people have announced via fms with:

hg fn-fmsread --listall

Repository hashes are used in the fms update trust map.

The default private key

When you run fn-setup, it creates a default SSK private key, which it stores inthe default_private_key parameter in your .infocalypse/infocalypse.ini file.

You can edit the config file to substitute any valid SSK private key you want.

If you specify an Insert URI without the key part for an infocalypse command the default private key is filled in for you. i.e

hg fn-create --uri USK@/test.R1/0

Inserts the local hg repository into a new USK in Freenet, using the private key in your config file.

USK <--> Directory mappings

The extension's commands 'remember' the insert and request repository USKs they were last run with when run again from the same directory.

This makes it unnecessary to retype cumbersome repository USK values once a repository has been successfully pulled or pushed from a directory.

Aggressive top key searching

fn-pull and fn-push have an --aggressive command line argument which causes them to search harder for the latest request URI.

This can be slow, especially if the USK index is much lower than the latest index in Freenet.

You will need to use it if you're not using FMS update notifications.

[TOC]

Basic Usage

Here are examples of basic commands.

Generating a new private key

You can generate an new private key with:

hg fn-genkey

This has no effect on the stored default private key.

Make sure to change the 'SSK' in the InsertURI to 'USK' when supplying the insert URI on the command line.

Creating a new repository

hg fn-create --uri USK@/test.R1/0

Inserts the local hg repository into a new USK in Freenet, using the privatekey in your config file. You can use a full insert URI value if you want.

If you see an "update -- Bundle too big to salt!" warning message when you run this command you should consider running
fn-reinsert --level 4.

Pushing to a repository

hg fn-push --uri USK@/test.R1/0

Pushes incremental changes from the local directory into an existing Infocalypse repository.

The <keypart>/test.R1/0 repository must already exist in Freenet.In the example above the default private key is used. You could have specified a full Insert URI. The URI must end in a number but the value doesn't matter because fn-push searches for the latest unused index.

You can ommit the --uri argument whenyou run from the same directory the fn-create (or a previous fn-push)was run from.

Pulling from a repository

hg fn-pull --uri <request uri>

pulls from an Infocalypse repository in Freenet intothe local repository.
Here's an example with a fully specified uri.

You can ommit the --uri argument whenyou run from the same directory a previous fn-pull was successfully run from.

For maximum reliability use the --aggressive argument.

[TOC]

Using FMS to send and receive update notifications

The extension can send and receive repository update notifications via FMS. It is highly recommended that you
setup this feature.

The update trust map

There's a trust map in the .infocalypse/infocalypse.ini config file which determines which fms ids can update the index values for which repositories. It is purely local and completely separate from the trust values which appear in the FMS web of trust.

The format is:
<number> = <fms_id>|<usk_hash0>|<usk_hash1>| ... |<usk_hashn>

The number value must be unique, but is ignored.

The fms_id values are the full FMS ids that you are trusting to update the repositories with the listed hashes.

The usk_hash* values are repository hashes.

Here's an example trust map config entry:

# Example .infocalypse snippet
[fmsread_trust_map]
1 = test0@adnT6a9yUSEWe5p8J-O1i8rJCDPqccY~dVvAmtMuC9Q|55833b3e6419
0 = djk@isFiaD04zgAgnrEC5XJt1i4IE7AkNPqhBG5bONi6Yks|be68e8feccdd|5582404a9124
2 = test1@SH1BCHw-47oD9~B56SkijxfE35M9XUvqXLX1aYyZNyA|fab7c8bd2fc3

You must update the trust map to enable index updating for repos other than the one this code lives in (be68e8feccdd). You can edit the config file directly if you want.

However, the easiest way to update the trust map is by using the--trust and --untrust options on fn-fmsread.

For example to trust falafel@IxVqeqM0LyYdTmYAf5z49SJZUxr7NtQkOqVYG0hvITwto notify you about changes to the repository with repo hash 2220b02cf7ee,type:

hg fn-fmsread --trust --hash 2220b02cf7ee --fmsid falafel@IxVqeqM0LyYdTmYAf5z49SJZUxr7NtQkOqVYG0hvITw

And to stop trusting that FMS id for updates to 2220b02cf7ee, you would type:

hg fn-fmsread --untrust --hash 2220b02cf7ee --fmsid falafel@IxVqeqM0LyYdTmYAf5z49SJZUxr7NtQkOqVYG0hvITw

To show the trust map type:

hg fn-fmsread --showtrust

Reading other people's notifications

hg fn-fmsread -v

Will read update notifications for all the repos in the trust map and locally cache the new latest index values. If you run with -vit prints a message when updates are available which weren't used because the sender(s) weren't in the trust map.

hg fn-fmsread --list

Displays announced repositories from fms ids that appear inthe trust map.

hg fn-fmsread --listall

Displays all announced repositories including ones from unknown fms ids.

Pulling an announced repository

You can use the --hash option with fn-pull to pull any repository you see in the fn-read --list or fn-read --listall lists.

For example to pull the latest version of the infocalypse extension code, cd to an empty directory and type:

hg inithg fn-pull --hash be68e8feccdd --aggressive

Posting your own notifications

hg fn-fmsnotify -v

Posts an update notification for the current repository to fms.

You MUST set the fms_id value in the config fileto your fms id for this to work.

Use --dryrun to double check before sending the actual fms message.

Use --announce at least once if you want your USK to show up in the fmsread --listall list.

By default notifications are written to and read from the infocalypse.notify fms group.

The read and write groups can be changed by editing the following variables in the config file:

fmsnotify_group = <group>
fmsread_groups = <group0>[|<group1>|...]

fms can have pretty high latency. Be patient. It may take hours (sometimes a day!) for your notification to appear. Don't send lots of redundant notifications.

[TOC]

Reinserting and 'sponsoring' repositories

hg fn-reinsert

will re-insert the bundles for the repository that was last pulled into the directory.

The exact behavior is determined by the level argument.

level:

• 1 - re-inserts the top key(s)
• 2 - re-inserts the top keys(s), graphs(s) and the most recent update.
• 3 - re-inserts the top keys(s), graphs(s) and all keys required to bootstrap the repo.

  This is the default level.

• 4 - adds redundancy for big (>7Mb) updates.
• 5 - re-inserts existing redundant big updates.

Levels 1 and 4 require that you have the privatekey for the repository. For other levels, the top key insert is skipped if you don't have the private key.

DO NOT use fn-reinsert if you're concerned about
correlation attacks. The risk is on the order of re-inserting a freesite, but may be worse if you use redundant(i.e. USK@<line noise>/name.R1/0) top keys.

[TOC]

Forking a repository onto a new USK

hg fn-copy --inserturi USK@/name_for_my_copy.R1/0

copies the Infocalypse repository which was fn-pull'd intothe local directory onto a new repository USK under your default private key. You can use a full insert URI if you want.

This only requires copying the top key data (a maximum of 2 SSK inserts).

[TOC]

Sharing private keys

It is possible for multiple people to collaborate anonymously over Freenet by sharing the private key to a single Infocalypse repository.

The FreeFAQ is an example of this technique.

Here are some things to keep in mind when sharing private keys.

• There is no (explict) key revocation in Freenet
  
  If you decide to share keys, you should generate a special key on a per repo basis with fn-genkey. There is no way to revoke a private key once it has been shared. This could be mitigated with an ad-hoc convention. e.g. if I find any file named USK@<public_key>/revoked.txt, I stop using the key.
• Non-atomic top key inserts
  
  Occasionally, you might end up overwriting someone elses commits because the FCP insert of the repo top key isn't atomic. I think you should be able to merge and re fn-push to resolve this. You can fn-pull a specific version of the repo by specify the full URI including the version number with --uri and including the --nosearch option.
• All contributors should be in the fn-fmsread trust map

[TOC]

Inserting a freesite

hg fn-putsite --index <n>

inserts a freesite based on the configuration inthe freesite.cfg file in the root of the repository.

Use:

hg fn-putsite --createconfig

to create a basic freesite.cfg file that you can modify. Look at the comments in it for an explanation of the supported parameters.

The default freesite.cfg file inserts using the same private key as the repo and a site name of 'default'. Editing the name is highly recommended.

You can use --key CHK@ to insert a test version of the site to a CHK key before writing to the USK.

Limitations:

• You MUST have fn-pushed the repo at least once in order to insert using the repo's private key. If you haven't fn-push'd you'll see this error: "You don't have the insert URI for this repo. Supply a private key with --key or fn-push the repo."
• Inserts all files in the site_dir directory in the freesite.cfg file. Run with --dryrun to make
  sure that you aren't going to insert stuff you don't want too.
• You must manually specify the USK edition you want to insert on. You will get a collision error
  if you specify an index that was already inserted.
• Don't use this for big sites. It should be fine for notes on your project. If you have lots of images
  or big binary files use a tool like jSite instead.
• Don't modify site files while the fn-putsite is running.

[TOC]

Risks

I don't believe that using this extension is significantly more dangerous that using any other piece of Freenet client code, but here is a list of the risks which come to mind:

• Freenet is beta software
  The authors of Freenet don't pretend to guarantee that it is free of bugs that could that could compromise your anonymity or worse.

  While written in Java, Freenet loads native code via JNI (FEC codecs, bigint stuff, wrapper, etc.) that makes it vulnerable to the same kinds of attacks as any other C/C++ code.

• FMS == anonymous software
  FMS is published anonymously on Freenet and it is written in C++ with dependencies on large libraries which could contain security defects.

  I personally build FMS from source and run it in a chroot jail.

  Somedude, the author of FMS, seems like a reputable guy and has conducted himself as such for more than a year.

• correlation attacks
  There is a concern that any system which inserts keys that can be predicted ahead of time could allow an attacker with control over many nodes in the network to eventually find the IP of your node.

  Any system which has this property is vulnerable. e.g. fproxy Freesite insertion,Freetalk, FMS, FLIP. This extension's optional use of
  redundant top keys may make it particularly vulnerable. If you are concerned don't use '.R1' keys.

  Running your node in pure darknet mode with trusted peers may somewhat reduce the risk of correlation attacks.

• Bugs in my code, Mercurial or Python
  I do my best but no one's perfect.

  There are lots of eyes over the Mercurial and Python source.

[TOC]

Advocacy

Here are some reasons why I think the Infocalypse 2.0 hg extension is better than
pyFreenetHg and
egit-freenet:

• Incremental

  You only need to insert/retrieve what has actually changed. Changes of up to 32kof compressed deltas can be fetched in as little as one SSK fetch and one CHK fetch.

• Redundant

  The top level SSK and the CHK with the representation of the repository state are inserted redundantly so there are no 'critical path' keys. Updates of up to ~= 7Mbare inserted redundantly by cloning the splitfile metadata at the cost of a single32k CHK insert.

• Re-insertable

  Anyone can re-insert all repository data except for the top level SSKs with a simple command (hg fn-reinsert). The repository owner can re-insert the top levelSSKs as well.

• Automatic rollups

  Older changes are automatically 'rolled up' into large splitfiles, such that the entire repository can almost always be fetched in 4 CHK fetches or less.

• Fails explictly

  REDFLAG DCI

[TOC]

Source Code

The authoritative repository for the extension's code is hosted in Freenet:

hg inithg fn-fmsread -vhg fn-pull --aggressive --debug --uri USK@kRM~jJVREwnN2qnA8R0Vt8HmpfRzBZ0j4rHC2cQ-0hw,2xcoQVdQLyqfTpF2DpkdUIbHFCeL4W~2X1phUYymnhM,AQACAAE/infocalypse.hgext.R1/41hg update

It is also mirrored on bitbucket.org:

hg clone http://bitbucket.org/dkarbott/infocalypse_hgext/

[TOC]

Fixes and version information

• hg version: c51dc4b0d282
  
  Fixed abort: <bundle_file> not found! problem on fn-pull when hg-git plugin was loaded.
• hg version: 0c5ce9e6b3b4
  
  Fixed intermittent stall when bootstrapping from an empty repo.
• hg version: 7f39b20500f0
  
  Fixed bug that kept fn-pull --hash from updating the initial USK index.
• hg version: 7b10fa400be1
  
  Added fn-fmsread --trust and --untrust and fn-pull --hash support.
  
  
  fn-pull --hash isn't really usable until 7f39b20500f0
• hg version: ea6efac8e3f6
  
  Fixed a bug that was causing the berkwood binary 1.3 Mercurial distribution
  (http://mercurial.berkwood.com/binaries/Mercurial-1.3.exe [HTTP Link!]) not to work.

[TOC]

Freenet-only links

This document is meant to inserted into Freenet.

It contains links (starting with 'CHK@' and 'USK@')to Freenet keys that will only work from within fproxy [HTTP link!].

You can find reasonably up to date version of this document on my freesite:

USK@-bk9znYylSCOEDuSWAvo5m72nUeMxKkDmH3nIqAeI-0,qfu5H3FZsZ-5rfNBY-jQHS5Ke7AT2PtJWd13IrPZjcg,AQACAAE/feral_codewright/15/infocalypse_howto.html

[TOC]

Contact

FMS:
djk@isFiaD04zgAgnrEC5XJt1i4IE7AkNPqhBG5bONi6Yks

I lurk on the freenet and fms boards.

If you really need to you can email me at d kar bott at com cast dot net but I prefer FMS.

freesite:
USK@-bk9znYylSCOEDuSWAvo5m72nUeMxKkDmH3nIqAeI-0,qfu5H3FZsZ-5rfNBY-jQHS5Ke7AT2PtJWd13IrPZjcg,AQACAAE/feral_codewright/15/

[TOC]

Install and setup infocalypse on GNU/Linux (script)

Install and setup infocalypse on GNU/Linux:

setup_infocalypse_on_linux.sh

Just download and run¹ it via

wget http://draketo.de/files/setup_infocalypse_on_linux.sh_1_0.txt
bash setup_infocalypse*

This script needs a running freenet node to work!

In-Freenet-link: CHK@RZjy7Whe3vT3aEdox3pEG4fRbmRGsyuybPPhdvr7MoQ,g8YZO1~FAJM5suS7Uch06ugblVPE4YJd1rl15DxAwkY,AAMC--8/setup_infocalypse_on_linux.sh

The script allows you to get and setup the infocalypse extension with a few keystrokes to be able to instantly use the Mercurial DVCS for decentral, anonymous code-sharing over freenet.

This gives you code hosting like a minimal version of BitBucket, Gitorious or GitHub but without the central control. Additionally the Sone plugin for freenet supplies anonymous communication and the site extension allows creating static sites with information about the repo, recent commits and such without the need of a dedicated hoster.

Basic Usage

Clone a repo into freenet with a new key:

hg clone localrepo USK@/repo

(Write down the insert key and request key after the upload! Localrepo is an existing Mercurial repository)

Clone a repo into or from freenet (respective key known):

hg clone localrepo freenet://USK@<insert key>/repo.R1/0
hg clone freenet://USK@<request key>/repo.R1/0 [localpath]

Push or pull new changes:

hg push freenet://USK@<insert key>/repo.R1/0
hg pull freenet://USK@<request key>/repo.R1/0

For convenient copy-pasting of freenet keys, you can omit the “freenet://” here, or use freenet:USK@… instead.

Also, as shown in the first example, you can let infocalypse generate a new key for your repo:

hg clone localrepo USK@/repo

mind the “USK@/” (slash after @ == missing key). Also see the missing .R1/0 after the repo-name and the missing freenet://. Being able to omit those on repository creation is just a convenience feature - but one which helps me a lot.

You can also add the keys to the <repo>/.hg/hgrc:

[paths]
example = freenet://USK@<request key>/repo.R1/0
example-push = freenet://USK@<insert key>/repo.R1/0
# here you need the freenet:// !

then you can simply use

hg push example-push

and

hg pull example

Contribute

This script is just a quick sketch, feel free to improve it and upload improved versions (for example with support for more GNU/Linux distros). If you experience any problems, please contact me! (i.e. write a comment)

If you want to contribute more efficiently to this script, get the repo via

hg clone freenet://USK@73my4fc2CLU3cSfntCYDFYt65R4RDmow3IT5~gTAWFk,Fg9EAv-Hut~9NCJKtGaGAGpsn1PjA0oQWTpWf7b1ZK4,AQACAAE/setup_infocalypse/1 

Then hack on it, commit and upload it again via

hg clone setup_infocalypse freenet://USK@/setup_infocalypse

Finally share the request URI you got.

Alternate repo: http://draketo.de/proj/setup_infocalypse

Spread Freenet: A call for action on identi.ca and twitter

“Daddy, where were you, when they took the freedom of the press away from the internet?” — Mike Godwin, Electronic Frontier Foundation

Reposted from Freetalk, the distributed pseudonymous forum in Freenet.

For all those among you, who use twitter¹ and/or identi.ca², this is a call to action.

Go to your identi.ca or twitter accounts and post about freenet. Tell us in 140 letters why freenet is your tool of choice, and remember to use the !freenet group (identi.ca) or #freenet hashtag (twitter), so we can resend your posts!

I use !freenet because we might soon need it as safe harbour to coordinate the fight against censorship → freenetproject.org !zensur — ArneBab

The broader story is the emerging concept of a right to freely exchange arbitrary data — Toad (the main freenet developer)

Background

There are still very many people out there who don’t know what freenet is. Just today a coder came into the #freenet IRC channel, asked what it did and learned that it already does everything he had thought about. And I still remember someone telling me “It would be cool if we had something like X-net from Cory Doctorow’s ‘Little Brother’” — he did not know that freenet already offers that with much improved security.

So we need to get the word out about freenet. And we have powerful word to choose from, beginning with Mike Godwin’s cite above but going much further. To just name a few buzz-words: Freenet is a crowdfunded distributed and censorship resistant freesoftware cloud publishing system. And different from info about corporate PR-powered projects, all these buzz words are true.

But to make us effective, we need to achieve critical mass. And to reach that, we need to coordinate and cross promote heavily.

Call to action

So I want to call to you to go to your identi.ca or twitter accounts and post about freenet. Tell us in 140 letters why freenet is your tool of choice, and remember to use the !freenet group or #freenet hashtag, so we can find and retweet your posts!

If you use identi.ca, join the !freenet group, so you get informed about new freenet-posts automatically.

We can make a difference, if we fight together.

And if you always wanted to get an identi.ca account, here’s the opportunity to get it and do something good at the same time :)

If you already have a twitter-account, you can connect your identi.ca account to your twitter account, then post to identi.ca and have your post forwarded to twitter automatically.

Additional info

Besides: My accounts are:

• http://twitter.com/ArneBab
• http://identi.ca/ArneBab

But no need to tell me your account and connect your Freetalk ID with it. Just use identi.ca or twitter and remember to tell your friends to talk about freenet, too (so we can’t find out who read this post and who decided to join in because he learned about the action from a friend).

As second line of defense, I also posted this message to my website and hereby allow anyone to reuse it in any form and under any license (up to the example tweets), so I can’t know who saw it here and who saw it elsewhere.

→ http://draketo.de/light/english/spread-freenet-a-call-to-action-on-twitter-and-identica

I hope I’ll soon see floods of entusiastic tweets and dents about Freenet!

Some example tweets and/or dents

I’ll gladly post and link yours here, if you allow it!

!Freenet: #crowdfunded distributed and censorship resistant !freesoftware cloud publishing → http://freenetproject.org — rightful buzz! — ArneBab

#imhappiestwhen when the internet is free. I hope it will remain so thanks to projects like #Freenet http://t.co/GMRXmDt — Gaming4JC

#freenet: freedom to publish that you may have to rely on, because censorship and ©ensorship are on the rise — Ixoliva

What can Freenet do well already?

this just happened to me in the #freenet IRC channel at freenode.net (somewhat edited):

• toad_¹: what can freenet do well already? [18:38]

• sharing and retrieving files asynchronously, freemail, IRC², publishing sites without need of a central server, sharing code repositories [18:39]

• I can simply go online, upload a file, send the key to a friend and go offline. the friend can then retrieve the file, even though I am already offline without needing a central server. [18:40]

• and nobody can eavesdrop.

• it might be kinda slow, but it actually makes it easy to publish stuff: via jSite, floghelper and others. [18:42]

• floghelper is cool: spam-resistant anonymous blogging without central server

• and freereader is, too (even though it needs lots of polish): forward RSS feeds into freenet

• you can actually exchange passwords in a safe way via freemail: anonymous email with an intergrated web-interface and imap access.

  • Justus and me coordinated the upload of the social networking site onto my FTP solely over freemail, and I did not have any fear of eavesdropping - different from any other mail I write. [18:44]

… I think I should store this conversation somewhere

which I hereby did - I hope you enjoyed this little insight into the #freenet channel :)

And if you grew interested, why not install freenet yourself? It only takes a few clicks via webstart and you’re part of the censorship-resistant web.

Wrapup: Make Sone scale - fast, anonymous, decentral microblogging over freenet

Sone¹ allows fast, identi.ca-style microblogging in Freenet. This is my wrapup on a discussion on the steps to take until Sone can become an integral part of Freenet.

Current state

• Is close to realtime.

• Downloads all IDs and all their posts and replies → polling which won’t scale; short term local breakage.

• Uploads all posts on every update → Can displace lots of content. Effective Size: X*M, X = revisions which did not drop out, M = total number of your messages. Long term self-DDoS of freenet.

Future

• Is close to realtime for those you follow and your usual discussion group.

• Uploads only recent posts directly and bundles older posts → much reduced storage need: Effective size: B * Z + Y*M; B = posts per bundle, Z = number of bundles which did not drop out, Y = numbers of not yet bundled messages; Z << Y, B << X, Y << X.

• Downloads only the ones you follow + ones you get told about. Telling others means that you need to include info about people you follow, because you only get information from them.

Telling others about replies, options

• Include all replies to anyone which I see in my own Sone → size rises massively, since you include all replies of all people you follow in your own Sone.

• Include all IDs from which you saw replies along with the people they replied to → needs to poll more IDs. Optionally forward that info for several hops → for efficient routing it needs knowledge about the full follower topology, which is a privacy risk.

• Discovering replies from people you don’t know yet: Add a WoT info: replies. Updated only when you reply to someone you did not reply to before. Poll people’s reply lists based on their WoT rating. Keep a list of people who answered one of your posts and poll these more often. Maybe poll people instantly who solve one of your captchas (your general captcha queue) → new users can enter quickly. When you solve captchas in WoT, preferably solve those from people you follow.
  → four ways to discover a reply:

  1. poll those you follow,
  2. poll the people who posted the latest replies to you (your usual discussion-group),
  3. poll those who solve one of your captchas (get new people in as fast as possible) and
  4. poll the replies-info from everyone with the polling frequency based on their WoT rating.

“regarding B.S. like SOPA, PIPA, … freenet seems like a good idea after all!”

“Some years ago, I had a look at freenet and wasn't really convinced, now I'm back - a lot has changed, it grew bigger and insanely fast (in freenet terms), like it a lot, maybe this time I'll keep it. Especially regarding B.S. like SOPA, PIPA and other internet-crippling movements, freenet seems like a good idea after all!”
— sparky in Sone

So, if you know freenet and it did not work out for you in the past, it might be time to give it another try: freenetproject.org

This quote just grabbed me, and sparky gave me permission to cite it.

A complete Mercurial branching strategy

This is a complete branching strategy for Mercurial with optional adaptions for maintaining multiple releases¹. It shows you all the actions you may need to take, except for those already described in the guide Mercurial in workflows.

For examples it uses the command-line UI, but it can easily be used with graphical Mercurial interfaces like TortoiseHG, too.

Summary

Firstoff, any model to be used by people should boil down to simple rules. Programming is complex enough without having to worry about elaborate branching rules. This model uses 3 simple rules:

(1) you do all the work on default² - except for hotfixes.

(2) on stable you only do hotfixes, merges for release³ and tagging for release. Only maintainers⁴ touch stable.

(3) you can use arbitrary feature-branches⁵, as long as you don’t call them default or stable. They always start at default (since you do all the work on default).

Diagram

To visualize the structure, here’s a 3-tiered diagram. To the left are the actions of developers (commits and feature branches) and in the center the tasks for maintainers (release and hotfix). The users to the right just use the stable branch.⁶

An overview of the branching strategy. Click the image to get the emacs org-mode ditaa-source.

Practial Actions

Now we can look at all the actions you will ever need to do in this model:⁷

• Initialize (only needed once)

  • create the repo: hg init reponame; cd reponame

  • first commit: (edit); hg ci -m "message"

  • create the stable branch and do the first release: hg branch stable; hg tag tagname; hg up default; hg merge stable; hg ci -m "merge stable into default: ready for more development"

• Regular development

  • commit changes: (edit); hg ci -m "message"

  • continue development after a release: hg update; (edit); hg ci -m "message"

• Feature Branches

  • start a larger feature: hg branch feature-x; (edit); hg ci -m "message"

  • continue with the feature: hg update feature-x; (edit); hg ci -m "message"

  • merge the feature: hg update default; hg merge feature-x; hg ci -m "merged feature x into default"

  • close and merge the feature when you are done: hg update feature-x; hg ci --close-branch -m "finished feature x"; hg update default; hg merge feature-x; hg ci -m "merged finished feature x into default"

• Tasks for Maintainers

  • apply a hotfix⁸: hg up stable; (edit); hg ci -m "message"; hg up default; hg merge stable; hg ci -m "merge stable into default: ready for more development"

  • do a release⁹: hg up stable; hg merge default; hg ci -m "merged default into stable for release" ; hg tag tagname; hg up default ; hg merge stable ; hg ci -m "merged stable into default: ready for more development"

Example

This is the output of a complete example run ¹⁰ of the branching model, including all complications you should ever hit.

We start with the full history. In the following sections, we will take it apart to see what the commands do. So just take a glance, take in the basic structure and then move on for the details.

hg log -G
@    changeset:   15:855a230f416f
|\   tag:         tip
| |  parent:      13:e7f11bbc756c
| |  parent:      14:79b616e34057
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:49 2013 +0100
| |  summary:     merged stable into default: ready for more development
| |
| o  changeset:   14:79b616e34057
|/|  branch:      stable
| |  parent:      7:e8b509ebeaa9
| |  parent:      13:e7f11bbc756c
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:48 2013 +0100
| |  summary:     merged default into stable for release
| |
o |    changeset:   13:e7f11bbc756c
|\ \   parent:      11:e77a94df3bfe
| | |  parent:      12:aefc8b3a1df2
| | |  user:        Arne Babenhauserheide <bab@draketo.de>
| | |  date:        Sat Jan 26 15:39:47 2013 +0100
| | |  summary:     merged finished feature x into default
| | |
| o |  changeset:   12:aefc8b3a1df2
| | |  branch:      feature-x
| | |  parent:      9:1dd6209b2a71
| | |  user:        Arne Babenhauserheide <bab@draketo.de>
| | |  date:        Sat Jan 26 15:39:46 2013 +0100
| | |  summary:     finished feature x
| | |
o | |  changeset:   11:e77a94df3bfe
|\| |  parent:      10:8c423bc00eb6
| | |  parent:      9:1dd6209b2a71
| | |  user:        Arne Babenhauserheide <bab@draketo.de>
| | |  date:        Sat Jan 26 15:39:45 2013 +0100
| | |  summary:     merged feature x into default
| | |
o | |  changeset:   10:8c423bc00eb6
| | |  parent:      8:dc61c2731eda
| | |  user:        Arne Babenhauserheide <bab@draketo.de>
| | |  date:        Sat Jan 26 15:39:44 2013 +0100
| | |  summary:     3
| | |
| o |  changeset:   9:1dd6209b2a71
|/ /   branch:      feature-x
| |    user:        Arne Babenhauserheide <bab@draketo.de>
| |    date:        Sat Jan 26 15:39:43 2013 +0100
| |    summary:     x
| |
o |  changeset:   8:dc61c2731eda
|\|  parent:      5:4c57fdadfa26
| |  parent:      7:e8b509ebeaa9
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:43 2013 +0100
| |  summary:     merged stable into default: ready for more development
| |
| o  changeset:   7:e8b509ebeaa9
| |  branch:      stable
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:42 2013 +0100
| |  summary:     Added tag v2 for changeset 089fb0af2801
| |
| o  changeset:   6:089fb0af2801
|/|  branch:      stable
| |  tag:         v2
| |  parent:      4:d987ce9fc7c6
| |  parent:      5:4c57fdadfa26
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:41 2013 +0100
| |  summary:     merge default into stable for release
| |
o |  changeset:   5:4c57fdadfa26
|\|  parent:      3:bc625b0bf090
| |  parent:      4:d987ce9fc7c6
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:40 2013 +0100
| |  summary:     merge stable into default: ready for more development
| |
| o  changeset:   4:d987ce9fc7c6
| |  branch:      stable
| |  parent:      1:a8b7e0472c5b
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:39 2013 +0100
| |  summary:     hotfix
| |
o |  changeset:   3:bc625b0bf090
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:38 2013 +0100
| |  summary:     2
| |
o |  changeset:   2:3e8df435bcb0
|\|  parent:      0:f97ea6e468a1
| |  parent:      1:a8b7e0472c5b
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:38 2013 +0100
| |  summary:     merged stable into default: ready for more development
| |
| o  changeset:   1:a8b7e0472c5b
|/   branch:      stable
|    user:        Arne Babenhauserheide <bab@draketo.de>
|    date:        Sat Jan 26 15:39:36 2013 +0100
|    summary:     Added tag v1 for changeset f97ea6e468a1
|
o  changeset:   0:f97ea6e468a1
   tag:         v1
   user:        Arne Babenhauserheide <bab@draketo.de>
   date:        Sat Jan 26 15:39:36 2013 +0100
   summary:     1

Action by action

Let’s take the log apart to show the actions contributors will do.

Initialize

Initializing and doing the first commit creates the first changeset:

o  changeset:   0:f97ea6e468a1
   tag:         v1
   user:        Arne Babenhauserheide <bab@draketo.de>
   date:        Sat Jan 26 15:39:36 2013 +0100
   summary:     1

Nothing much to see here.

Commands:

hg init test-branch; cd test-branch
(edit); hg ci -m "message"

Stable branch and first release

With the stable branch and first release, we add the tagging commit and merge back into default:

o    changeset:   2:3e8df435bcb0
|\   parent:      0:f97ea6e468a1
| |  parent:      1:a8b7e0472c5b
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:38 2013 +0100
| |  summary:     merged stable into default: ready for more development
| |
| o  changeset:   1:a8b7e0472c5b
|/   branch:      stable
|    user:        Arne Babenhauserheide <bab@draketo.de>
|    date:        Sat Jan 26 15:39:36 2013 +0100
|    summary:     Added tag v1 for changeset f97ea6e468a1
|
o  changeset:   0:f97ea6e468a1
   tag:         v1
   user:        Arne Babenhauserheide <bab@draketo.de>
   date:        Sat Jan 26 15:39:36 2013 +0100
   summary:     1

Mind the tag field which is now shown in changeset 0 and the branchname for changeset 1.

Commands:

hg branch stable
hg tag tagname
hg up default
hg merge stable
hg ci -m "merged stable into default: ready for more development"`

Further development

Now we just chuck along. The one commit shown here could be an arbitrary number of commits.

o    changeset:   3:bc625b0bf090
|    user:        Arne Babenhauserheide <bab@draketo.de>
|    date:        Sat Jan 26 15:39:38 2013 +0100
|    summary:     2
|  
o    changeset:   2:3e8df435bcb0
|\   parent:      0:f97ea6e468a1
| |  parent:      1:a8b7e0472c5b
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:38 2013 +0100
| |  summary:     merged stable into default: ready for more development

Commands:

(edit)
hg ci -m "message"

Hotfix

If a hotfix has to be applied to the release out of order, we just update to the stable branch, apply the hotfix and then merge the stable branch into default¹¹. This gives us changesets 4 for the hotfix and 5 for the merge (2 and 3 are shown as reference).

o    changeset:   5:4c57fdadfa26
|\   parent:      3:bc625b0bf090
| |  parent:      4:d987ce9fc7c6
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:40 2013 +0100
| |  summary:     merge stable into default: ready for more development
| |
| o  changeset:   4:d987ce9fc7c6
| |  branch:      stable
| |  parent:      1:a8b7e0472c5b
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:39 2013 +0100
| |  summary:     hotfix
| |
o |  changeset:   3:bc625b0bf090
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:38 2013 +0100
| |  summary:     2
| |
o |  changeset:   2:3e8df435bcb0
|\|  parent:      0:f97ea6e468a1
| |  parent:      1:a8b7e0472c5b
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:38 2013 +0100
| |  summary:     merged stable into default: ready for more development

Commands:

hg up stable
(edit)
hg ci -m "message"
hg up default
hg merge stable
hg ci -m "merge stable into default: ready for more development"    

Regular release

To do a regular release, we just merge the default branch into the stable branch and tag the merge. Then we merge stable back into default. This gives us changesets 6 to 8¹².

o    changeset:   8:dc61c2731eda
|\   parent:      5:4c57fdadfa26
| |  parent:      7:e8b509ebeaa9
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:43 2013 +0100
| |  summary:     merged stable into default: ready for more development
| |
| o  changeset:   7:e8b509ebeaa9
| |  branch:      stable
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:42 2013 +0100
| |  summary:     Added tag v2 for changeset 089fb0af2801
| |
| o  changeset:   6:089fb0af2801
|/|  branch:      stable
| |  tag:         v2
| |  parent:      4:d987ce9fc7c6
| |  parent:      5:4c57fdadfa26
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:41 2013 +0100
| |  summary:     merge default into stable for release
| |
o |  changeset:   5:4c57fdadfa26
|\|  parent:      3:bc625b0bf090
| |  parent:      4:d987ce9fc7c6
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:40 2013 +0100
| |  summary:     merge stable into default: ready for more development

Commands:

hg up stable
hg merge default
hg ci -m "merge default into stable for release"
hg tag tagname
hg up default
hg merge stable
hg ci -m "merged stable into default: ready for more development"

Feature branches

Now we want to do some larger development, so we use a feature branch. The one feature-commit shown here (x) could be an arbitrary number of commits, and as long as you stay in your branch, the development of your colleagues will not disturb your own work. Once the feature is finished, we merge it into default. That gives us changesets 9 to 13.

o    changeset:   13:e7f11bbc756c
|\   parent:      11:e77a94df3bfe
| |  parent:      12:aefc8b3a1df2
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:47 2013 +0100
| |  summary:     merged finished feature x into default
| |
| o  changeset:   12:aefc8b3a1df2
| |  branch:      feature-x
| |  parent:      9:1dd6209b2a71
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:46 2013 +0100
| |  summary:     finished feature x
| |
o |  changeset:   11:e77a94df3bfe
|\|  parent:      10:8c423bc00eb6
| |  parent:      9:1dd6209b2a71
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:45 2013 +0100
| |  summary:     merged feature x into default
| |
o |  changeset:   10:8c423bc00eb6
| |  parent:      8:dc61c2731eda
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:44 2013 +0100
| |  summary:     3
| |
| o  changeset:   9:1dd6209b2a71
|/   branch:      feature-x
|    user:        Arne Babenhauserheide <bab@draketo.de>
|    date:        Sat Jan 26 15:39:43 2013 +0100
|    summary:     x
|  
o    changeset:   8:dc61c2731eda
|\   parent:      5:4c57fdadfa26
| |  parent:      7:e8b509ebeaa9
| |  user:        Arne Babenhauserheide <bab@draketo.de>
| |  date:        Sat Jan 26 15:39:43 2013 +0100
| |  summary:     merged stable into default: ready for more development

Commands:

• Start the feature

  hg branch feature-x 
  (edit)
  hg ci -m "message"
  

• Do an intermediate commit on default

  hg update default
  (edit)
  hg ci -m "message"
  

• Continue working on the feature

  hg update feature-x
  (edit)
  hg ci -m "message"
  

• Merge the feature

  hg update default
  hg merge feature-x
  hg ci -m "merged feature x into default"`
  

• Close and merge a finished feature

  hg update feature-x
  hg ci --close-branch -m "finished feature x"
  hg update default; hg merge feature-x
  hg ci -m "merged finished feature x into default"
  

Note: Closing the feature branch hides that branch in the output of hg branches (except when using --closed) to make the repository state lean and simple while still keeping the feature branch information in history. It shows your collegues, that they no longer have to keep the feature in mind as soon as they merge the most recent changes from the default branch into their own feature branches.

Note: To make the final merge of your feature into default easier, you can regularly merge the default branch into the feature branch.

Note: We use feature branches to ensure that new clones start at a revision which other developers can directly use. With bookmarks you could get trapped on a feature-head which might not be merged to default for quite some time. For more reasons, see the bookmarks footnote.

The final action is a regular merge to stable to get into a state from which we could safely do a release. Since we already showed how to do that, we are finished here.

Extensions

This realizes the successful Git branching model¹³ with Mercurial while maintaining one release at any given time.

If you have special needs, this model can easily be extended to fullfill your requirements. Useful extensions include:

• multiple releases - if you need to provide maintenance for multiple releases side-by-side.
• grafted micro-releases - if you need to segment the next big changes into smaller releases while leaving out some potentially risky changes.
• explicit review - if you want to ensure that only reviewed changes can get into a release, while making it possible to leave out some already reviewed changes from the next releases. Review gets decoupled from releasing.

All these extensions are orthogonal, so you can use them together without getting side-effects.

Multiple maintained releases

To use the branching model with multiple simultaneously maintained releases, you only need to change the hotfix procedure: When applying a hotfix, you go back to the old release with hg update tagname, fix there, add a new tag for the fixed release and then update to the next release. There you merge the new fix-release and do the same for all other releases. If the most recent release is not the head of the stable branch, you also merge into stable. Then you merge the stable branch into default, as for a normal hotfix.¹⁴

With this merge-chain you don’t need special branches for releases, but all changesets are still clearly recorded. This simplification over git is a direct result of having real anonymous branching in Mercurial.

hg update release-1.0
(edit)
hg ci -m "message"
hg tag release-1.1
hg update release-2.0
hg merge release-1.1
hg ci -m "merged changes from release 1.1"
hg tag release-2.1
… and so on

In the Diagram this just adds a merge path from the hotfix to the still maintained releases.

An overview of the branching strategy with maintained releases. Click the image to get the emacs org-mode ditaa-source.

Graft changes into micro-releases

If you need to test parts of the current development in small chunks, you can graft micro releases. In that case, just update to stable and merge the first revision from default, whose child you do not want, and graft later changes¹⁵.

Example for the first time you use micro-releases¹⁶:

You have changes 1, 2, 3, 4 and 5 on default. First you want to create a release which contains 1 and 4, but not 2, 3 or 5.

hg update 1
hg branch stable
hg graft 4

As usual tag the release and merge stable back into default:

hg tag rel-14 
hg update default
hg merge stable
hg commit -m "merge stable into default. ready for more development"

Example for the second and subsequent releases:

Now you want to release the change 2 and 5, but you’re still not ready to release 3. So you merge 2 and graft 5.

hg update stable
hg merge 2
hg commit -m "merge all changes until 2 from default"
hg graft 5

As usual tag the release and finally merge stable back into default:

hg tag rel-1245 
hg update default
hg merge stable
hg commit -m "merge stable into default. ready for more development"

The history now looks like this¹⁷:

@    merge stable into default. ready for more development (default)
|\
| o  Added tag rel-1245 for changeset 4e889731c6ca (stable)
| |
| o  5 (stable)
| |
| o    merge all changes until 2 from default (stable)
| |\
o---+  merge stable into default. ready for more development (default)
| | |
| | o  Added tag rel-14 for changeset cc2c95dd3f27 (stable)
| | |
| | o  4 (stable)
| | |
o | |  5 (default)
| | |
o | |  4 (default)
| | |
o | |  3 (default)
|/ /
o /  2 (default)
|/
o  1 (default)
|
o  0 (default)

In the Diagram this just adds graft commits to stable:

An overview of the branching strategy with grafted micro-releases. Click the image to get the emacs org-mode ditaa-source.

Grafted micro-releases add another layer between development and releases. They can be necessary in cases where testing requires actually deploying a release, as for example in Freenet.

Explicit review branch

If you want to add a separate review stage, you can use a review branch¹⁸¹⁹ into which you only merge or graft reviewed changes. The review branch then acts as a staging area for all changes which might go into a release.

To use this extension of the branching model, just create a branch on default called review in which you merge or graft reviewed changes. The first time you do that, you update to the first commit whose children you do not want to include. Then create the review branch with hg branch review and use hg graft REV to pull in all changes you want to include.

On subsequent reviews, you just update to review with hg update nextrelease, merge the first revision which has a child you do not want with hg merge REV and graft additional later changes with hg graft REV.

In both cases you create the release by merging the review branch into stable or grafting changes from it as you would do it for micro-releases.

A special condition when using a review branch is that you always have to merge hotfixes into the review branch, too, because the review branch does not automatically contain all changes from the default branch.

In the Diagram this just adds the review branch between default and stable instead of the release merge. Also it adds the hotfix merge to the review branch.

An overview of the branching strategy with a review branch. Click the image to get the emacs org-mode ditaa-source.

Simple Summary

We now have nice graphs, examples, potential extensions and so on. But since this strategy uses Mercurial instead of git, we don’t actually need all the graphics, descriptions and branch categories in the git version - or in this post.

Instead we can boil all of this down to 3 simple rules:

(1) you do all the work on default - except for hotfixes.

(2) on stable you only do hotfixes, merges for release and tagging for release. Only maintainers touch stable.

(3) you can use arbitrary feature-branches, as long as you don’t call them default or stable. They always start at default (since you do all the work on default).

They are the rules you already know from the starting summary.

That’s it. Happy hacking!

A short introduction to Mercurial with TortoiseHG (GNU/Linux and Windows)

Downloading the Repository

After installing TortoiseHG, you can download a repository to your computer by right-clicking in a folder and selecting the menu "TortoiseHG" and then "Clone" in there (currently you still need Windows for that - all other dialogs can be evoked in GNU/Linux on the commandline via "hgtk").

Right-Click menu, Windows:

Create Clone, GNU/Linux:

In the dialog you just enter the url of the repository, for example:

http://www.bitbucket.org/ArneBab/md-esw-2009

(that's also the address of the repository in the internet - just try clicking the link.

When you log in to bitbucket.org you will find a clone-address directly on the site. You can also use that clone address to upload changes (it contains your login-name, and I can give you "push" access on that site).

Workflow with TortoiseHG

This gives you two basic abilities:

• Save and view changes locally, and
• synchronize changes with others.

(I assume that part of what I say is redundant, but I'd rather write a bit too much than omit a crucial bit)

To save changes, you can simlply select "HG Commit" in the right-click-menu. If some of your files aren't known to HG yet (the box before the file isn't ticked), you have to add them (tick the box) to be able to commit them.

To go back to earlier changes, you can use "Checkout Revision" in the "TortoiseHG" menu. In that dialog you can then select the revision you want to see and use the icon on the upper left to get all files to that revision.

You can synchronize by right-clicking in the folder and selecting "Synchronize" in the "TortoiseHG" menu (inside the right-click menu). In the opening dialog you can "push" (upload changes - arrow up with the bar above it), "pull" (download changes to your computer - arrow down with bar below), and check what you would pull or push (arrows iwthout bars). I thing that using dialog will soon became second nature for you, too :)

Have fun with TortoiseHG! :) - Arne

PS: There's also a longer intro to TortoiseHG and an overview to DVCS.

PPS: md-esw-2009 is a repository in which Baddok and I planned a dual-gm roleplaying session Mechanical Dream.

PPPS: There's also a german version of this article on my german pages.

Basic usecases for DVCS: Workflow Failures

Update (2013-04-18): In #mercurial @ irc.freenode.net there were discussions yesterday for improving the help output if you do not have your username setup, yet.

LC_ALL=C
LANG=C
PS1="$"
rm -rf /tmp/gitflow > /dev/null
mkdir -p /tmp/gitflow > /dev/null
cd /tmp/gitflow > /dev/null
# init the repo
git init orig  > /dev/null
cd orig > /dev/null
echo 1 > 1
# add a commit
git add 1 > /dev/null
git config user.name upstream > /dev/null
git config user.email up@stream > /dev/null
git commit -m 1 > /dev/null
# checkout another branch but master. YES, YOU SHOULD DO THAT on the shared repo. We’ll see later, why.
git checkout -b never-pull-this-temporary-useless-branch master 2> /dev/null
cd .. > /dev/null
echo # purely cosmetic and implementation detail: this adds a new line to the output
ls

wolf, n.:
    A man who knows all the ankles.
arne@fluss ~/.emacs.d/private/journal $ arne@fluss ~/.emacs.d/private/journal $ $$$$$$$$$$$$$$$$
orig

git --version

git version 1.8.1.5

git clone orig mine
echo $ ls
ls

Cloning into 'mine'...
done.
$ ls
mine  orig

cd mine
echo 2 > 1
git commit -m "hack"

$# On branch master
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)

modified:   1
no changes added to commit (use "git add" and/or "git commit -a")

git config user.email "contributor"
git config user.name "con@tribut.or"

git commit -m "hack"

On branch master
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)

modified:   1
no changes added to commit (use "git add" and/or "git commit -a")

git add 1
git commit -m "hack"

[master aba911a] hack
 1 file changed, 1 insertion(+), 1 deletion(-)

git push

warning: push.default is unset; its implicit value is changing in
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the current behavior after the default changes, use:

  git config --global push.default matching

To squelch this message and adopt the new behavior now, use:

  git config --global push.default simple

See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)

Counting objects: 5, done.
(1/3)   
Writing objects:  66% (2/3)   
Writing objects: 100% (3/3)   
Writing objects: 100% (3/3), 222 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To /tmp/gitflow/orig
master

cd ..
git clone mine test
cd test
# setup the user locally again. Normally you do not need that again, since you’d use --global.
git config user.email "contributor" 
git config user.name "con@tribut.or"
# hack and commit
echo test > 1
git add 1
echo # cosmetic
git commit -m "change to test" >/dev/null
# (run the tests)

git push

warning: push.default is unset; its implicit value is changing in
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the current behavior after the default changes, use:

  git config --global push.default matching

To squelch this message and adopt the new behavior now, use:

  git config --global push.default simple

See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)

Counting objects: 5, done.
(1/3)   
Writing objects:  66% (2/3)   
Writing objects: 100% (3/3)   
Writing objects: 100% (3/3), 234 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: error: refusing to update checked out branch: refs/heads/master        
remote: error: By default, updating the current branch in a non-bare repository        
remote: error: is denied, because it will make the index and work tree inconsistent        
remote: error: with what you pushed, and will require 'git reset --hard' to match        
remote: error: the work tree to HEAD.        
remote: error:         
remote: error: You can set 'receive.denyCurrentBranch' configuration variable to        
remote: error: 'ignore' or 'warn' in the remote repository to allow pushing into        
remote: error: its current branch; however, this is not recommended unless you        
remote: error: arranged to update its work tree to match what you pushed in some        
remote: error: other way.        
remote: error:         
remote: error: To squelch this message and still keep the default behaviour, set        
remote: error: 'receive.denyCurrentBranch' configuration variable to 'refuse'.        
To /tmp/gitflow/mine
master (branch is currently checked out)
error: failed to push some refs to '/tmp/gitflow/mine'

git checkout -b testing master
git push ../mine testing

Switched to a new branch 'testing'
Counting objects: 5, done.
(1/3)   

cd ../mine
git merge testing
git push

Updating aba911a..820dea8
Fast-forward
 1 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
warning: push.default is unset; its implicit value is changing in
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the current behavior after the default changes, use:

  git config --global push.default matching

To squelch this message and adopt the new behavior now, use:

  git config --global push.default simple

See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)

Counting objects: 5, done.
(1/3)   
Writing objects:  66% (2/3)   
Writing objects: 100% (3/3)   
Writing objects: 100% (3/3), 234 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To /tmp/gitflow/orig
master

LC_ALL=C
LANG=C
PS1="$"
rm -rf /tmp/hgflow > /dev/null
mkdir -p /tmp/hgflow > /dev/null
cd /tmp/hgflow > /dev/null
# init the repo
hg init orig  > /dev/null
cd orig > /dev/null
echo 1 > 1 > /dev/null
# add a commit
hg add 1 > /dev/null
hg commit -u upstream -m 1 > /dev/null
cd .. >/dev/null
echo # purely cosmetic and implementation detail: this adds a new line to the output
ls

The most happy marriage I can imagine to myself would be the union
of a deaf man to a blind woman.
        -- Samuel Taylor Coleridge
arne@fluss ~/.emacs.d/private/journal $ arne@fluss ~/.emacs.d/private/journal $ $$$$$$$$$$$$
orig

hg --version

Mercurial Distributed SCM (version 2.5.2)
(see http://mercurial.selenic.com for more information)

Copyright (C) 2005-2012 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

hg clone orig mine
echo $ ls
ls

updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ ls
mine  orig

cd mine
echo 2 > 1
echo
# I disable the username to show the problem
hg --config ui.username= commit -m "hack" 

$
$abort: no username supplied (see "hg help config")

hg help config | head -n 30 | grep -B 3 -A 1 per-repository

These files do not exist by default and you will have to create the
    appropriate configuration files yourself: global configuration like the
USERPROFILE%\mercurial.ini" or
HOME/.hgrc" and local configuration is put into the per-repository
/.hg/hgrc" file.

echo '[ui]
username = "contributor"' >> .hg/hgrc

hg commit -m "hack"

hg push

pushing to /tmp/hgflow/orig
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files

cd ..
hg clone mine test
cd test
# setup the user locally again. Normally you do not need that again, since you’d use --global.
echo '[ui]
username = "contributor"' >> .hg/hgrc
# hack and commit
echo test > 1
echo # cosmetic
hg commit -m "change to test"
# (run the tests)

updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$$> $$$

hg push

pushing to /tmp/hgflow/mine
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files

cd ../mine
hg push

pushing to /tmp/hgflow/orig
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files

Creating nice logs with revsets in Mercurial

In the mercurial list Stanimir Stamenkov asked how to get rid of intermediate merges in the log to simplify reading the history (and to not care about missing some of the details).

Update: Since Mercurial 2.4 you can simply use
hg log -Gr "branchpoint()"

I did some tests for that and I think the nicest representation I found is this:

hg log -Gr "(all() - merge()) or head()"

The result

It showed that in the end the revisions converged again - and it shows the actual states of the development.

$ hg log -Gr "(all() - merge()) or head()"

@    Änderung:        7:52fe4a8ec3cc
|\   Marke:           tip
| |  Vorgänger:       6:7d3026216270
| |  Vorgänger:       5:848c390645ac
| |  Nutzer:          Arne Babenhauserheide <bab@draketo.de>
| |  Datum:           Tue Aug 14 15:09:54 2012 +0200
| |  Zusammenfassung: merge
| |
| \
| |\
| | o  Änderung:        3:55ba56aa8299
| | |  Vorgänger:       0:385d95ab1fea
| | |  Nutzer:          Arne Babenhauserheide <bab@draketo.de>
| | |  Datum:           Tue Aug 14 15:09:40 2012 +0200
| | |  Zusammenfassung: 4
| | |
| o |  Änderung:        2:b500d0a90d40
| |/   Vorgänger:       0:385d95ab1fea
| |    Nutzer:          Arne Babenhauserheide <bab@draketo.de>
| |    Datum:           Tue Aug 14 15:09:39 2012 +0200
| |    Zusammenfassung: 3
| |
o |  Änderung:        1:8cc66166edc9
|/   Nutzer:          Arne Babenhauserheide <bab@draketo.de>
|    Datum:           Tue Aug 14 15:09:38 2012 +0200
|    Zusammenfassung: 2
|
o  Änderung:        0:385d95ab1fea
   Nutzer:          Arne Babenhauserheide <bab@draketo.de>
   Datum:           Tue Aug 14 15:09:38 2012 +0200
   Zusammenfassung: 1

Even shorter, but not quite correct

The shortest representation is without the heads, though. It does not represent the current state of development if the last commit was a merge or if some branches were not merged. Otherwise it is equivalent.

$ hg log -Gr "(all() - merge())"

o  Änderung:        3:55ba56aa8299
|  Vorgänger:       0:385d95ab1fea
|  Nutzer:          Arne Babenhauserheide <bab@draketo.de>
|  Datum:           Tue Aug 14 15:09:40 2012 +0200
|  Zusammenfassung: 4
|
| o  Änderung:        2:b500d0a90d40
|/   Vorgänger:       0:385d95ab1fea
|    Nutzer:          Arne Babenhauserheide <bab@draketo.de>
|    Datum:           Tue Aug 14 15:09:39 2012 +0200
|    Zusammenfassung: 3
|
| o  Änderung:        1:8cc66166edc9
|/   Nutzer:          Arne Babenhauserheide <bab@draketo.de>
|    Datum:           Tue Aug 14 15:09:38 2012 +0200
|    Zusammenfassung: 2
|
o  Änderung:        0:385d95ab1fea
   Nutzer:          Arne Babenhauserheide <bab@draketo.de>
   Datum:           Tue Aug 14 15:09:38 2012 +0200
   Zusammenfassung: 1

The basic log For reference

The vanilla-log looks like this:

$ hg log -G

@    Änderung:        7:52fe4a8ec3cc
|\   Marke:           tip
| |  Vorgänger:       6:7d3026216270
| |  Vorgänger:       5:848c390645ac
| |  Nutzer:          Arne Babenhauserheide <bab@draketo.de>
| |  Datum:           Tue Aug 14 15:09:54 2012 +0200
| |  Zusammenfassung: merge
| |
| o    Änderung:        6:7d3026216270
| |\   Vorgänger:       2:b500d0a90d40
| | |  Vorgänger:       4:8dbc55213c9f
| | |  Nutzer:          Arne Babenhauserheide <bab@draketo.de>
| | |  Datum:           Tue Aug 14 15:09:45 2012 +0200
| | |  Zusammenfassung: merged 4
| | |
o | |  Änderung:        5:848c390645ac
|\| |  Vorgänger:       3:55ba56aa8299
| | |  Vorgänger:       2:b500d0a90d40
| | |  Nutzer:          Arne Babenhauserheide <bab@draketo.de>
| | |  Datum:           Tue Aug 14 15:09:43 2012 +0200
| | |  Zusammenfassung: merged 2
| | |
+---o  Änderung:        4:8dbc55213c9f
| | |  Vorgänger:       3:55ba56aa8299
| | |  Vorgänger:       1:8cc66166edc9
| | |  Nutzer:          Arne Babenhauserheide <bab@draketo.de>
| | |  Datum:           Tue Aug 14 15:09:41 2012 +0200
| | |  Zusammenfassung: merged 1
| | |
o | |  Änderung:        3:55ba56aa8299
| | |  Vorgänger:       0:385d95ab1fea
| | |  Nutzer:          Arne Babenhauserheide <bab@draketo.de>
| | |  Datum:           Tue Aug 14 15:09:40 2012 +0200
| | |  Zusammenfassung: 4
| | |
| o |  Änderung:        2:b500d0a90d40
|/ /   Vorgänger:       0:385d95ab1fea
| |    Nutzer:          Arne Babenhauserheide <bab@draketo.de>
| |    Datum:           Tue Aug 14 15:09:39 2012 +0200
| |    Zusammenfassung: 3
| |
| o  Änderung:        1:8cc66166edc9
|/   Nutzer:          Arne Babenhauserheide <bab@draketo.de>
|    Datum:           Tue Aug 14 15:09:38 2012 +0200
|    Zusammenfassung: 2
|
o  Änderung:        0:385d95ab1fea
   Nutzer:          Arne Babenhauserheide <bab@draketo.de>
   Datum:           Tue Aug 14 15:09:38 2012 +0200
   Zusammenfassung: 1

Creating the test repo

To create the test repo, I just used a few short loops in the shell:

hg init test ; cd test 
for i in 1 2 3 4; do echo $i > $i ; hg ci -Am "$i"; hg up -r -$i; done
for i in 1 2 3 4; do echo $i > $i ; hg ci -Am "$i"; hg up -r -$i; hg merge $i ; hg ci -m "merged $i"; done
for i in $(hg heads --template "{node} ") ; do hg merge $i ; hg ci -m "merge"; done

Better representations?

Do you have better representations for viewing convoluted history?

PS: Yes, you can rewrite history, but that’s a really bad idea if you have many people who closely interact and publish early and often.

Gentoo live ebuild for Mercurial

We (nelchael and me) just finished a live ebuild for Mercurial which allows to conveniently track the main (mpm) repo of Mercurial in Gentoo.

To use the ebuild, just add

=dev-util/mercurial-9999 **  

to your package.keywords and emerge mercurial (again).

It took us a while since we had to revise the Mercurial eclass to always build Mercurial live packages from their Mercurial repository and nelchael took the chance to completely overhaul the eclass.

If you're interested in the details, please have a look at the ebuild and the eclass as well as the tracking bug.

To use the eclass in an ebuild, just add inherit mercurial at the beginning of the ebuild and set EHG_REPO_URI to the correct repository URI. If you need to share a single repository between several ebuilds, set EHG_PROJECT to the project name in all of them.

Have fun with Mercurial!

Learning Mercurial in Workflows

The official workflow guide for Mercurial, mirrored from mercurial.selenic.com/guide. License: GPLv2 or later.

It delves into nonlinear history and merging right from the beginning and uses only features you get without activating extensions. Due to this it offers efficient and safe workflows without danger of losing already committed work.

• Basic workflows
• Log keeping
• Lone developer with nonlinear history
• Separate features
• Sharing changes
• Summary
• Advanced workflows
• Backing out bad revisions
• Collaborative feature development
• Tagging revisions
• Removing history
• Summary
• More complex workflows
• License

With Mercurial you can use a multitude of different workflows. This page shows some of them, including their use cases. It is intended to make it easy for beginners of version tracking to get going instantly and learn completely incrementally. It doesn't explain the concepts used, because there are already many other great resources doing that, for example the wiki and the hgbook.

If you want a more exhaustive tutorial with the basics, please have a look at the Tutorial in the Mercurial Wiki. For a really detailed and very nice to read description of Mercurial, please have a look at Mercurial: The Definitive Guide.

Basic workflows

We go from simple to more complex workflows. Those further down build on previous workflows.

Log keeping

Use Case

The first workflow is also the easiest one: You want to use Mercurial to be able to look back when you did which changes.

This workflow only requires an installed Mercurial and write access to some file storage (you almost definitely have that :) ). It shows the basic techniques for more complex workflows.

Workflow

Prepare Mercurial

As first step, you should teach Mercurial your name. For that you open the file ~/.hgrc (or mercurial.ini in your home directory for Windows) with a text-editor and add the ui section (user interaction) with your username:

[ui]
username = Mr. Johnson <johnson@smith.com>

Initialize the project

Now you add a new folder in which you want to work:

$ hg init project

Add files and track them

$ cd project
$ (add files)
$ hg add
$ hg commit
(enter the commit message)

$ cd project
$ hg init

$ hg add file0* file10 file11 file12

Save changes

$ (do some changes)

see which files changed, which have been added or removed, and which aren't tracked yet

$ hg status

see the exact changes

$ hg diff

commit the changes.

$ hg commit

now an editor pops up and asks you for a commit message. Upon saving and closing the editor, your changes have been stored by Mercurial.

Move and copy files

When you copy or move files, you should tell Mercurial to do the copy or move for you, so it can track the relationship between the files.

Remember to commit after moving or copying. From the basic commands only commit creates a new revision

$ hg cp original copy
$ hg commit
(enter the commit message)
$ hg mv original target
$ hg commit
(enter the commit message)

Now you have two files, "copy" and "target", and Mercurial knows how they are related.

Check your history

$ hg log

This prints a list of changesets along with their date, the user who committed them (you) and their commit message.

To see a certain revision, you can use the -r switch (--revision). To also see the diff of the displayed revisions, there's the -p switch (--patch)

$ hg log -p -r 3

Lone developer with nonlinear history

Use case

The second workflow is still very easy: You're a lone developer and you want to use Mercurial to keep track of your own changes.

It works just like the log keeping workflow, with the difference that you go back to earlier changes at times.

To start a new project, you initialize a repository, add your files and commit whenever you finished a part of your work.

Also you check your history from time to time, so see how you progressed.

Workflow

Basics from log keeping

Init your project, add files, see changes and commit them.

$ hg init project
$ cd project
$ (add files)
$ hg add # tell Mercurial to track all files
$ (do some changes)
$ hg diff # see changes
$ hg commit # save changes
$ hg cp # copy files or folders
$ hg mv # move files or folders
$ hg log # see history

Seeing an earlier revision

Different from the log keeping workflow, you'll want to go back in history at times and do some changes directly there, for example because an earlier change introduced a bug and you want to fix it where it occurred.

To look at a previous version of your code, you can use update. Let's assume that you want to see revision 3.

$ hg update 3

Now your code is back at revision 3, the fourth commit (Mercurial starts counting at 0).
To check if you're really at that revision, you can use identify -n.

$ hg identify -n

To update to the most recent revision, you can use "tip" as revision name.

$ hg update tip

Fixing errors in earlier revisions

When you find a bug in some earlier revision you have two options: either you can fix it in the current code, or you can go back in history and fix the code exactly where you did it, which creates a cleaner history.

To do it the cleaner way, you first update to the old revision, fix the bug and commit it. Afterwards you merge this revision and commit the merge. Don't worry, though: Merging in mercurial is fast and painless, as you'll see in an instant.

Let's assume the bug was introduced in revision 3.

$ hg update 3
$ (fix the bug)
$ hg commit

Now the fix is already stored in history. We just need to merge it with the current version of your code.

$ hg merge

If there are conflicts use hg resolve - that's also what merge tells you to do in case of conflicts.

First list the files with conflicts

$ hg resolve --list

Then resolve them one by one. resolve attempts the merge again

$ hg resolve conflicting_file
(fix it by hand, if necessary)

Mark the fixed file as resolved

$ hg resolve --mark conflicting_file

Commit the merge, as soon as you resolved all conflicts. This step is also necessary when there were no conflicts!

$ hg commit

At this point, your fix is merged with all your other work, and you can just go on coding. Additionally the history shows clearly where you fixed the bug, so you'll always be able to check where the bug was.

So now you can initialize repositories, save changes, update to previous changes and develop in a nonlinear history by committing in earlier changesets and merging the changes into the current code.

Separate features

Use Case

At times you'll be working on several features in parallel. If you want to avoid mixing incomplete code versions, you can create clones of your local repository and work on each feature in its own code directory.

After finishing your feature you then pull it back into your main directory and merge the changes.

Workflow

Work in different clones

First create the feature clone and do some changes

$ hg clone project feature1
$ cd feature1
$ (do some changes and commits)

Now check what will come in when you pull from feature1, just like you can use diff before committing. The respective command for pulling is incoming

$ cd ../project
$ hg incoming ../feature1

If you like the changes, you pull them into the project

$ hg pull ../feature1

Now you have the history of feature1 inside your project, but the changes aren't yet visible. Instead they are only stored inside a ".hg" directory of the project (more information on the store).

If you didn't do any changes in the project, while you were working on feature1, you can just update to tip (hg update tip), but it is more likely that you'll have done some other changes in between changes. In that case, it's time for merging.

Merge feature1 into the project code

$ hg merge

If there are conflicts use hg resolve - that's also what merge tells you to do in case of conflicts. After you merge, you have to commit explicitly to make your merge final

$ hg commit
(enter commit message, for example "merged feature1")

You can create an arbitrary number of clones and also carry them around on USB sticks. Also you can use them to synchronize your files at home and at work, or between your desktop and your laptop.

Rollback mistakes

Now you can work on different features in parallel, but from time to time a bad commit might sneak in. Naturally you could then just go back one revision and merge the stray error, keeping all mistakes out of the merged revision. However, there's an easier way, if you realize your error before you do another commit or pull: rollback.

Rolling back means undoing the last operation which added something to your history.

Imagine you just realized that you did a bad commit - for example you didn't see a spelling error in a label. To fix it you would use

hg rollback

And then redo the commit

hg commit -m "message"

If you can use the command history of your shell and you added the previous message via commit -m "message", that following commit just means two clicks on the arrow-key "up" and one click on "enter".

Though it changes your history, rolling back doesn't change your files. It only undoes the last addition to your history.

But beware, that a rollback itself can't be undone. If you rollback and then forget to commit, you can't just say "give me my old commit back". You have to create a new commit.

Sharing changes

Use Case

Now we go one step further: You are no longer alone, and you want to share your changes with others and include their changes.

The basic requirement for that is that you have to be able to see the changes of others.

Mercurial allows you to do that very easily by including a simple webserver from which you can pull changes just as you can pull changes from local clones.

Workflow

Using the builtin webserver

This is the easiest way to quickly share changes.

First the one who wants to share his changes creates the webserver

$ hg serve

Now all others can point their browsers to his IP address (for example 192.168.178.100) at port 8000. They will then see all his history there and can decide if they want to pull his changes.

$ firefox http://192.168.178.100:8000

If they decide to include the changes, they just pull from the same URL

$ hg pull http://192.168.178.100:8000

At this point you all can work as if you had pulled from a local repository. All the data is now in your individual repositories and you can merge the changes and work with them without needing any connection to the served repository.

Sending changes by email

Often you won't have direct access to the repository of someone else, be it because he's behind a restrictive firewall, or because you live in different timezones. You might also want to keep your changes confidential and prefer internal email (if you want additional protection, you can also encrypt the emails, for example with GnuPG).

In that case, you can easily export your changes as patches and send them by email.

Another reason to send them by email can be that your policy requires manual review of the changes when the other developers are used to reading diffs in emails. I'm sure you can think of more reasons.

Sending the changes via email is pretty straightforward with Mercurial. You just export your changes and attach (or copy paste) it in your email. Your colleagues can then just import them.

First check which changes you want to export

$ cd project
$ hg log

We assume that you want to export changeset 3 and 4

$ hg export 3 > change3.diff
$ hg export 4 > change4.diff

Now attach them to an email and your colleagues can just run import on both diffs to get your full changes, including your user information.

To be careful, they first clone their repository to have an integration directory as sandbox

$ hg clone project integration
$ cd integration
$ hg import change3.diff
$ hg import change4.diff

That's it. They can now test your changes in feature clones. If they accept them, they pull the changes into the main repository

$ cd ../project
$ hg pull ../integration

$ hg bundle --base FIRST_REVISION_TO_BUNDLE changes.bundle

$ hg pull path/to/changes.bundle

Using a shared repository

Sending changes by email might be the easiest way to reach people when you aren't yet part of the regular development team, but it creates additional workload: You have to bundle the changes, send mails and then import the bundles manually. Luckily there's an easier way which works quite well: The shared push repository.

Till now we transferred all changes either via email or via pull. Now we use another option: pushing. As the name suggests it's just the opposite of pulling: You push your changes into another repository.

But to make use of it, we first need something we can push to.

By default hg serve doesn't allow pushing, since that would be a major security hole. You can allow pushing in the server, but that's no solution when you live in different timezones, so we'll go with another approach here: Using a shared repository, either on an existing shared server or on a service like BitBucket. Doing so has a bit higher starting cost and takes a bit longer to explain, but it's well worth the effort spent.

If you want to use an existing shared server, you can use serve there and allow pushing. Also there are some other nice ways to allow pushing to a Mercurial repository, including simple access via SSH.

Otherwise you first need to setup a BitBucket Account. Just signup at BitBucket. After signing up (and login) hover your mouse over "Repositories". There click the item at the bottom of the opening dialog which say "Create new".

Give it a name and a description. If you want to keep it hidden from the public, select "private"

$ firefox http://bitbucket.org

Now your repository is created and you see instructions for pushing to it. For that you'll use a command similar to the following (just with a different URL)

$ hg push https://bitbucket.org/ArneBab/hello/

(Replace the URL with the URL of your created repository. If your username is "Foo" and your repository is named "bar", the URL will be https://bitbucket.org/Foo/bar/)

Mercurial will ask for your BitBucket name and password, then push your code.

Voilà, your code is online.

Now it's time to tell all your colleagues to sign up at BitBucket, too.

After that you can click the "Admin" tab of your created repository and add the usernames of your colleagues on the right side under "Permission: Writers". Now they are allowed to push code to the repository.

(If you chose to make the repository private, you'll need to add them to "Permission: Readers", too)

If one of you now wants to publish changes, he'll simply push them to the repository, and all others get them by pulling.

Publish your changes

$ hg push https://bitbucket.org/ArneBab/hello/

Pull others changes into your local repository

$ hg pull https://bitbucket.org/ArneBab/hello/

People who join you in development can also just clone this repository, as if one of you were using hg serve

$ hg clone https://bitbucket.org/ArneBab/hello/ hello

That local repository will automatically be configured to pull/push from/to the online repository, so new contributors can just use hg push and hg pull without an URL.

Summary

Now let's take a step back and look where we are.

With the commands you already know, a bit reading of hg help <command> and some evil script-fu you can already do almost everything you'll ever need to do when working with source code history. So from now on almost everything is convenience, and that's a good thing.

First this is good, because it means, that you can now use most of the concepts which are utilized in more complex workflows.

Second it aids you, because convenience lets you focus on your task instead of focusing on your tool. It helps you concentrate on the coding itself. Still you can always go back to the basics, if you want to.

A short summary of what you can do which can also act as a short check, if you still remember the meaning of the commands.

create a project

$ hg init project
$ cd project
$ (add some files)
$ hg add
$ hg commit
(enter the commit message)

do nonlinear development

$ (do some changes)
$ hg commit
(enter the commit message)
$ hg update 0
$ (do some changes)
$ hg commit
(enter the commit message)
$ hg merge
$ (optionally hg resolve)
$ hg commit
(enter the commit message)

use feature clones

$ cd ..
$ hg clone project feature1
$ cd feature1
$ (do some changes)
$ hg commit
(enter the commit message)
$ cd ../project
$ hg pull ../feature1

share your repository via the integrated webserver

$ hg serve &
$ cd ..
$ hg clone http://127.0.0.1:8000 project-clone

export changes to files

$ cd project-clone
$ (do some changes)
$ hg commit
(enter the commit message)
$ hg export tip > ../changes.diff

import changes from files

$ cd ../project
$ hg import ../changes.diff

pull changes from a served repository (hg serve still runs on project)

$ cd ../feature1
$ hg pull http://127.0.0.1:8000

Use shared repositories on BitBucket

$ (setup bitbucket repo)
$ hg push https://bitbucket.org/USER/REPO
(enter name and password in the prompt)
$ hg pull https://bitbucket.org/USER/REPO

Let's move on towards useful features and a bit more advanced workflows.

Advanced workflows

Backing out bad revisions

Use Case

When you routinely pull code from others, it can happen that you overlook some bad change. As soon as others pull that change from you, you have little chance to get completely rid of it.

To resolve that problem, Mercurial offers you the backout command. Backing out a change means, that you tell Mercurial to create a commit which reverses the bad change. That way you don't get rid of the bad code in history, but you can remove it from new revisions.

Workflow

Let's assume the bad change was revision 3, and you already have one more revision in your
repository. To remove the bad code, you can just backout of it. This creates a new
change which reverses the bad change. After backing out, you can then merge that new change
into the current code.

$ hg backout 3
$ hg merge
(potentially resolve conflicts)
$ hg commit
(enter commit message. For example: "merged backout")

That's it. You reversed the bad change. It's still recorded that it was once there (following the principle "don't rewrite history, if it's not really necessary"), but it doesn't affect future code anymore.

Collaborative feature development

Now that you can share changes and reverse them if necessary, you can go one step further: Using Mercurial to help in coordinating the coding.

The first part is an easy way to develop features together, without requiring every developer to keep track of several feature clones.

Use Case

When you want to split your development into several features, you need to keep track of who works on which feature and where to get which changes.

Mercurial makes this easy for you by providing named branches. They are a part of the main repository, so they are available to everyone involved. At the same time, changes committed on a certain branch don't get mixed with the changes in the default branch, so features are kept separate, until they get merged into the default branch.

Workflow

When someone in your group wants to start coding on a feature without disturbing the others, he can create a named branch and commit there. When someone else wants to join in, he just updates to the branch and commits away. As soon as the feature is finished, someone merges the named branch into the default branch.

Working in a named branch

Create the branch

$ hg branch feature1
(do some changes)
$ hg commit
(write commit message)

Update into the branch and work in it

$ hg update feature1
(do some changes)
$ hg commit
(write commit message)

Now you can commit, pull, push and merge (and anything else) as if you were working in a separate repository. If the history of the named branch is linear and you call "hg merge", Mercurial asks you to specify an explicit revision, since the branch in which you work doesn't have anything to merge.

Merge the named branch

When you finished the feature, you merge the branch back into the default branch.

$ hg update default
$ hg merge feature1
$ hg commit
(write commit message)

And that's it. Now you can easily keep features separate without unnecessary bookkeeping.

Tagging revisions

Use Case

Since you can now code separate features more easily, you might want to mark certain revisions as fit for consumption (or similar). For example you might want to mark releases, or just mark off revisions as reviewed.

For this Mercurial offers tags. Tags add a name to a revision and are part of the history. You can tag a change years after it was committed. The tag includes the information when it was added, and tags can be pulled, pushed and merged just like any other committed change.

Workflow

Let's assume you want to give revision 3 the name "v0.1".

Add the tag

$ hg tag -r 3 v0.1

See all tags

$ hg tags

When you look at the log you'll now see a line in changeset 3 which marks the Tag. If someone wants to update to the tagged revision, he can just use the name of your tag

$ hg update v0.1

Now he'll be at the tagged revision and can work from there.

Removing history

Use Case

At times you will have changes in your repository, which you really don't want in it.

There are many advanced options for removing these, and most of them use great extensions (Mercurial Queues is the most often used one), but in this basic guide, we'll solve the problem with just the commands we already learned. But we'll use an option to clone which we didn't yet use.

This workflow becomes inconvenient when you need to remove changes, which are buried below many new changes. If you spot the bad changes early enough, you can get rid of them without too much effort, though.

Workflow

Let's assume you want to get rid of revision 2 and the highest revision is 3.

The first step is to use the "--rev" option to clone: Create a clone which only contains the changes up to the specified revision. Since you want to keep revision 1, you only clone up to that

$ hg clone -r 1 project stripped

Now you can export the change 3 from the original repository (project) and import it into the stripped one

$ cd project
$ hg export 3 > ../changes.diff
$ cd ../stripped
$ hg import ../changes.diff

If a part of the changes couldn't be applied, you'll see that part in *.rej files. If you have *.rej files, you'll have to include or discard changes by hand

$ cat *.rej
(apply changes by hand)
$ hg commit
(write commit message)

That's it. hg export also includes the commit message, date, committer and similar metadata, so you are already done.

Summary

So now you can work with Mercurial in private, and also share your changes in a multitude of ways.

Additionally you can remove bad changes, either by creating a change in the repository which reverses the original change, or by really rewriting history, so it looks like the change never occurred.

And you can separate the work on features in a single repository by using named branches and add tags to revisions which are visible markers for others and can be used to update to the tagged revisions.

With this we can conclude our practical guide.

More Complex Workflows

If you now want to check some more complex workflows, please have a look at the general workflows wikipage.

To deepen your understanding, you should also check the basic concept overview.

Have fun with Mercurial!

Mercurial for two Programmers who are (mostly) new to SCM

Written in the Mercurial mailing list

Hi Bernard,

Am Dienstag 03 Februar 2009 20:19:14 schrieb ... ...:
> Most of the docs I can find seem to assume the reader is familiar with
> existing software developemnt tools and methodologies.
>
> This is not the case for me.

It wasn't for me either, and I can assure you that using Mercurial becomes
natural quite quickly.

> Now, I need to coordinate with a second (also SCM clueless) programmer.
...
> I envision us both working the main trunk for many small day-to-day
> changes, and our own isolated repo for larger additions that we will each
> be working on.

I don't know about a HOWTO, but I can give you a short description about basic
usage and the workflow I'd use:

Basic usage

• Just commit as you'd have done in SVN via "hg commit".
• To get changes from others, do "hg pull -u".
  The "-u" says 'update my files'.
• If you already committed and then pull changes from someone else, you merge
  the changes with yours via "hg merge". Merging is quite painless in Mercurial, so you can easily do it often.
• Once you want to share your changes, do "hg push".
  Should that complain about "adding heads", pull and merge, then do the push again. If you really want to create new remote heads, you can use "hg push -f".

Workflow

• Firstoff: Create a main repository you both can push changes to. If you have ssh access to a shared machine, that's as simple as creating a repository on that machine via "hg init project".
• Now both of you clone from that repository via
  hg clone ssh://USER@ADDRESS:path/to/project project

  (ADDRESS can be either a host or an IP).

  That's your repository for the small day to day changes.

• If you want to do bigger changes, you create a feature clone via
  hg clone project feature1

  In that clone you simply work, pull and commit as usual, but you only push after you finished the feature.

  Once you finished the feature, you push the changes from the feature clone via "hg push" in feature1 (which gets them into your main working clone) and then push then onward into the shared repository.

That's it - or rather that's what I'd do. It might be right for you, too, and
if it isn't, don't be shy of experimenting. As long as you have a backup clone
lying around (for example cloned to a USB stick via "hg clone project
path/to/stick/project"), you can't do too much damage :)

I hope I could provide a bit of help :)

Mercurial Workflow: Feature seperation via named branches

Also published on Mercurials Workflows wikipage. Originally written for PyHurd: Python bindings for the GNU Hurd.

For Whom?

If you

1. want to develop features collaboratively and you want to be able to see later for which feature a given change was added or
2. want to do changes concurrently which would likely affect each other negatively while they are not finished, but which need to be developed in a group with minimal overhead,

then this workflow might be right for you.

Note: If you have a huge number of small features (2000 and upwards), the number of persistent named branches can create certain performance problems. For features which need no collaboration or need only a few commits, this workflow also has much unnecessary overhead. It is best used for features which will be developed side by side with default for some time (and many commits), so tracking the default branch against the feature is relevant. To mark single-commit features as belonging to a feature, just use the commit message.

Note: The difference between Mercurial named branches and git branches is that git branches don’t stay in history. They don’t allow you to find out later in which branch a certain commit was added. If you want git-style branching, just use bookmarks.

What you need

Just vanilla Mercurial.

Workflow

The workflow is 6-stepped:

1. create the new feature,
2. Implement and share,
3. merge other changes into it,
4. merge stable features,
5. close finished features and
6. reopen features.

Let’s see the steps in detail.

1. New feature

First start a new branch with the name of the feature (starting from default).

hg branch feature-x
# do some changes
hg commit -m "Started implemented feature-x"

2. Implement and share

Then commit away and push whenever you finish something which might be of interest to others, regardless how marginal.

You can push to a shared repository, or to your own clone or even send the changes via email to other contributors (for example via the mailbomb extension).

3. Merge in default

Merge changes in the default branch into your feature as often as possible to reduce the work necessary when you want to merge the feature later on.

hg update feature-x
hg merge default
hg commit -m "merged default into feature-x"

4. Merge stable features

When your feature is stable, merge it into default.

hg update default
hg merge feature-x
hg commit -m "merged feature-x"

5. Close the branch when it’s done

And when the feature needs no more work, close the branch.

# start from default, automatic when using a fresh clone
hg update default
hg branch feature-x
# do some changes
hg commit -m "started feature X" 
hg push 

# commit and push as you like

hg update default
hg merge feature-x
hg ci -m "merged feature X into default"
hg commit --close-branch -m "finished feature X"

This hides the branch from the output of hg branches, so you don’t clutter your history.

6. Reopen the feature

To improve a feature after it was officially closed, first merge default into the feature branch (to get it up to date), then work just as if you had started it.

hg up feature-x
hg merge default
hg ci -m "merged default into feature X"
# commit, push, repeat, finish

Generally merge default into your feature as often as possible.

Epilog

If this workflow helps you, I’d be glad to hear from you!

Test of the hg evolve extension for easier upstreaming

Table of Contents

• 1 Rationale
• 2 Tests
  • 2.1 Init
  • 2.2 Prepare
  • 2.3 Amend
  • 2.4 …together
    • 2.4.1 Setup
    • 2.4.2 Fix my side
    • 2.4.3 Be a nice neighbor
  • 2.5 …safely
  • 2.6 Fold
  • 2.7 Split
  • 2.8 …as afterthought
• 3 Conclusion

# Tests for refactoring history with the evolve extension
export LANG=C # to get rid of localized strings
export PS1="$ "
rm -r testmy testother testpublic

# Initialize the test repo
hg init testpublic # a public repo
hg init testmy # my repo
hg init testother # other repo
# make the two private repos nonpublishing
for i in my other
  do echo "[ui]
username = $i
[phases]
publish = False" > test${i}/.hg/hgrc
done

cd testmy
echo "Hello World" > hello.txt
hg ci -Am "Hello World"
hg log -G
cd ..

adding hello.txt
@  changeset:   0:c19ed5b17f4f
   tag:         tip
   user:        my
   date:        Sat Jan 12 00:17:40 2013 +0100
   summary:     Hello World

cd testmy
sed -i s/World/Evoluton/ hello.txt
hg ci -m "Hello Evolution"
echo
hg log -G
cat hello.txt
# FIX this up
sed -i s/Evoluton/Evolution/ hello.txt
hg amend -m "Hello Evolution" # pass the message explicitely again to avoid having the editor pop up
echo
hg log -G
cd ..

@  changeset:   1:83a5e89adea6
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:17:41 2013 +0100
|  summary:     Hello Evolution
|
o  changeset:   0:c19ed5b17f4f
   user:        my
   date:        Sat Jan 12 00:17:40 2013 +0100
   summary:     Hello World
Hello Evoluton

@  changeset:   3:129d59901401
|  tag:         tip
|  parent:      0:c19ed5b17f4f
|  user:        my
|  date:        Sat Jan 12 00:17:42 2013 +0100
|  summary:     Hello Evolution
|
o  changeset:   0:c19ed5b17f4f
   user:        my
   date:        Sat Jan 12 00:17:40 2013 +0100
   summary:     Hello World

cd testmy
# Now we add the bad change
echo "Wishes:
- The Solek wants Action
- The Judicator wants Action

" >> plan.txt
hg ci -Am "What the players want"
# show what we did
echo
hg log -G -r tip
# and the good change
echo "Places: 
- The village
- The researchers cave
" >> plan.txt
hg ci -m "The places"
echo
hg log -G -r 1:
cd ..

  adding plan.txt

@  changeset:   4:b170dda0a4a7
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:17:44 2013 +0100
|  summary:     What the players want
|
 
@  changeset:   5:2a37053027cc
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:17:45 2013 +0100
|  summary:     The places
|
o  changeset:   4:b170dda0a4a7
|  user:        my
|  date:        Sat Jan 12 00:17:44 2013 +0100
|  summary:     What the players want
|
o  changeset:   3:129d59901401
|  parent:      0:c19ed5b17f4f
|  user:        my
|  date:        Sat Jan 12 00:17:42 2013 +0100
|  summary:     Hello Evolution
|

hg -R testother pull -u testmy
hg -R testother log -G -r 1:

pulling from testmy
requesting all changes
adding changesets
adding manifests
adding file changes
added 4 changesets with 4 changes to 2 files
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
@  changeset:   3:2a37053027cc
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:17:45 2013 +0100
|  summary:     The places
|
o  changeset:   2:b170dda0a4a7
|  user:        my
|  date:        Sat Jan 12 00:17:44 2013 +0100
|  summary:     What the players want
|
o  changeset:   1:129d59901401
|  user:        my
|  date:        Sat Jan 12 00:17:42 2013 +0100
|  summary:     Hello Evolution
|

cd testother
echo "People:
- The Lost
- The Specter
" >> plan.txt
hg ci -m "The people"
echo
hg log -G -r 1:
cd ..

 
@  changeset:   4:65cc97fc774a
|  tag:         tip
|  user:        other
|  date:        Sat Jan 12 00:17:48 2013 +0100
|  summary:     The people
|
o  changeset:   3:2a37053027cc
|  user:        my
|  date:        Sat Jan 12 00:17:45 2013 +0100
|  summary:     The places
|
o  changeset:   2:b170dda0a4a7
|  user:        my
|  date:        Sat Jan 12 00:17:44 2013 +0100
|  summary:     What the players want
|
o  changeset:   1:129d59901401
|  user:        my
|  date:        Sat Jan 12 00:17:42 2013 +0100
|  summary:     Hello Evolution
|

cd testmy
hg up -r -2
sed -i "s/The Solek wants Action/The Solek wants emotionally intense situations/" plan.txt
hg amend -m "The wishes of the players"
hg log -G -r 1:
cd ..

1 files updated, 0 files merged, 0 files removed, 0 files unresolved
1 new unstable changesets
@  changeset:   7:86e7a5305c9e
|  tag:         tip
|  parent:      3:129d59901401
|  user:        my
|  date:        Sat Jan 12 00:17:50 2013 +0100
|  summary:     The wishes of the players
|
| o  changeset:   5:2a37053027cc
| |  user:        my
| |  date:        Sat Jan 12 00:17:45 2013 +0100
| |  summary:     The places
| |
| x  changeset:   4:b170dda0a4a7
|/   user:        my
|    date:        Sat Jan 12 00:17:44 2013 +0100
|    summary:     What the players want
|
o  changeset:   3:129d59901401
|  parent:      0:c19ed5b17f4f
|  user:        my
|  date:        Sat Jan 12 00:17:42 2013 +0100
|  summary:     Hello Evolution
|

cd testmy
hg evolve
hg log -G -r 1:
cd ..

move:[5] The places
atop:[7] The wishes of the players
merging plan.txt
@  changeset:   8:0980732d20e0
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:17:45 2013 +0100
|  summary:     The places
|
o  changeset:   7:86e7a5305c9e
|  parent:      3:129d59901401
|  user:        my
|  date:        Sat Jan 12 00:17:50 2013 +0100
|  summary:     The wishes of the players
|
o  changeset:   3:129d59901401
|  parent:      0:c19ed5b17f4f
|  user:        my
|  date:        Sat Jan 12 00:17:42 2013 +0100
|  summary:     Hello Evolution
|

hg -R testother pull testmy
hg -R testother log -G

pulling from testmy
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 2 changes to 1 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
1 new unstable changesets
o  changeset:   6:0980732d20e0
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:17:45 2013 +0100
|  summary:     The places
|
o  changeset:   5:86e7a5305c9e
|  parent:      1:129d59901401
|  user:        my
|  date:        Sat Jan 12 00:17:50 2013 +0100
|  summary:     The wishes of the players
|
| @  changeset:   4:65cc97fc774a
| |  user:        other
| |  date:        Sat Jan 12 00:17:48 2013 +0100
| |  summary:     The people
| |
| x  changeset:   3:2a37053027cc
| |  user:        my
| |  date:        Sat Jan 12 00:17:45 2013 +0100
| |  summary:     The places
| |
| x  changeset:   2:b170dda0a4a7
|/   user:        my
|    date:        Sat Jan 12 00:17:44 2013 +0100
|    summary:     What the players want
|
o  changeset:   1:129d59901401
|  user:        my
|  date:        Sat Jan 12 00:17:42 2013 +0100
|  summary:     Hello Evolution
|
o  changeset:   0:c19ed5b17f4f
   user:        my
   date:        Sat Jan 12 00:17:40 2013 +0100
   summary:     Hello World

hg -R testother evolve

nothing to evolve here
(1 troubled changesets, do you want --any ?)

hg -R testother evolve --any
hg -R testother log -G -r 1:

move:[4] The people
atop:[6] The places
merging plan.txt
@  changeset:   7:058175606243
|  tag:         tip
|  user:        other
|  date:        Sat Jan 12 00:17:48 2013 +0100
|  summary:     The people
|
o  changeset:   6:0980732d20e0
|  user:        my
|  date:        Sat Jan 12 00:17:45 2013 +0100
|  summary:     The places
|
o  changeset:   5:86e7a5305c9e
|  parent:      1:129d59901401
|  user:        my
|  date:        Sat Jan 12 00:17:50 2013 +0100
|  summary:     The wishes of the players
|
o  changeset:   1:129d59901401
|  user:        my
|  date:        Sat Jan 12 00:17:42 2013 +0100
|  summary:     Hello Evolution
|

cd testother
hg up > /dev/null
echo "current phase"
hg phase .
hg push ../testpublic
echo "phase after publishing"
hg phase .
cd ..

current phase
7: draft
pushing to ../testpublic
searching for changes
adding changesets
adding manifests
adding file changes
added 5 changesets with 5 changes to 2 files
phase after publishing
7: public

cd testother
hg amend -m "change published history"
# change to draft
hg phase -fd .
hg phase .
# now we could amend, but that would defeat the point of this section, so we go to public again.
hg phase -p .
cd ..

abort: can not rewrite immutable changeset 058175606243
7: draft

cd testmy
echo "Scenes:" >> plan.txt
hg ci -m "we need scenes"

echo "- Lost appears" >> plan.txt
hg ci -m "scene"
echo "- People vanish" >> plan.txt
hg ci -m "scene"
echo "- Portals during dreamtime" >> plan.txt
hg ci -m "scene"
echo
hg log -G -r 9:
cd ..

@  changeset:   12:fbcce7ad7369
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:18:06 2013 +0100
|  summary:     scene
|
o  changeset:   11:189c0362a80f
|  user:        my
|  date:        Sat Jan 12 00:18:05 2013 +0100
|  summary:     scene
|
o  changeset:   10:715a31ac9dee
|  user:        my
|  date:        Sat Jan 12 00:18:05 2013 +0100
|  summary:     scene
|
o  changeset:   9:dfa4c226150b
|  user:        my
|  date:        Sat Jan 12 00:18:05 2013 +0100
|  summary:     we need scenes
|

cd testmy
# hg fold -r "-1:-4"
# hg log -G -r 9:
cd ..

cd testmy
sed -i "s/The Judicator wants Action/The Judicator wants Action - portals/" plan.txt
sed -i "s/The village/The village - lost, vanish, portals/" plan.txt
hg ci -m "Apply Scenes to people and places."
echo
hg log -G -r 12:
cd ..

@  changeset:   13:5c83a3540c37
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:18:10 2013 +0100
|  summary:     Apply Scenes to people and places.
|
o  changeset:   12:fbcce7ad7369
|  user:        my
|  date:        Sat Jan 12 00:18:06 2013 +0100
|  summary:     scene
|

cd testmy
hg uncommit --all # to undo all changes, not just those for specified files
hg diff
sed -i "s/The village - lost, vanish, portals/The village/" plan.txt
hg amend -m "Apply scenes to wishes"
sed -i "s/The village/The village - lost, vanish, portals/" plan.txt
hg commit -m "Apply scenes to places"
echo
hg log -G -r 12:
cd ..

new changeset is empty
(use "hg kill ." to remove it)
diff --git a/plan.txt b/plan.txt
--- a/plan.txt
+++ b/plan.txt
@@ -1,10 +1,10 @@
 Wishes:
 - The Solek wants emotionally intense situations
-- The Judicator wants Action
+- The Judicator wants Action - portals
 
 
 Places: 
-- The village
+- The village - lost, vanish, portals
 - The researchers cave
 
 Scenes:

@  changeset:   17:f8cc86f681ac
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:18:13 2013 +0100
|  summary:     Apply scenes to places
|
o  changeset:   16:6c8918a352e2
|  parent:      12:fbcce7ad7369
|  user:        my
|  date:        Sat Jan 12 00:18:12 2013 +0100
|  summary:     Apply scenes to wishes
|
o  changeset:   12:fbcce7ad7369
|  user:        my
|  date:        Sat Jan 12 00:18:06 2013 +0100
|  summary:     scene
|

cd testmy
hg pull ../testother
hg merge  --tool internal:merge tip # the new head from our co-gamemaster
# fix the conflicts 
sed -i "s/<<<.*local//" plan.txt
sed -i "s/====.*/\n/" plan.txt
sed -i "s/>>>.*other//" plan.txt
# mark them as solved.
hg resolve -m
hg commit -m "merge people"
echo
hg log -G -r 12:
cd ..

pulling from ../testother
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
(run 'hg heads .' to see heads, 'hg merge' to merge)
merging plan.txt
warning: conflicts during merge.
merging plan.txt incomplete! (edit conflicts, then use 'hg resolve --mark')
0 files updated, 0 files merged, 0 files removed, 1 files unresolved
use 'hg resolve' to retry unresolved file merges or 'hg update -C .' to abandon

@    changeset:   19:8bf8d55739fa
|\   tag:         tip
| |  parent:      17:f8cc86f681ac
| |  parent:      18:058175606243
| |  user:        my
| |  date:        Sat Jan 12 00:18:16 2013 +0100
| |  summary:     merge people
| |
| o  changeset:   18:058175606243
| |  parent:      8:0980732d20e0
| |  user:        other
| |  date:        Sat Jan 12 00:17:48 2013 +0100
| |  summary:     The people
| |
o |  changeset:   17:f8cc86f681ac
| |  user:        my
| |  date:        Sat Jan 12 00:18:13 2013 +0100
| |  summary:     Apply scenes to places
| |
o |  changeset:   16:6c8918a352e2
| |  parent:      12:fbcce7ad7369
| |  user:        my
| |  date:        Sat Jan 12 00:18:12 2013 +0100
| |  summary:     Apply scenes to wishes
| |
o |  changeset:   12:fbcce7ad7369
| |  user:        my
| |  date:        Sat Jan 12 00:18:06 2013 +0100
| |  summary:     scene
| |

cd testmy
sed -i "s/The Solek wants emotionally intense situations/The Solek wants emotionally intense situations | specter, Lost/" plan.txt
sed -i "s/Lost appears/Lost appears | Lost/" plan.txt
sed -i "s/People vanish/People vanish | Specter/" plan.txt
hg commit -m "apply people to wishes, places and scenes"
echo
hg log -G -r 19:
cat plan.txt
cd ..

@  changeset:   20:c00aa6f24c3f
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:18:18 2013 +0100
|  summary:     apply people to wishes, places and scenes
|
o    changeset:   19:8bf8d55739fa
|\   parent:      17:f8cc86f681ac
| |  parent:      18:058175606243
| |  user:        my
| |  date:        Sat Jan 12 00:18:16 2013 +0100
| |  summary:     merge people
| |
Wishes:
- The Solek wants emotionally intense situations | specter, Lost
- The Judicator wants Action - portals


Places: 
- The village - lost, vanish, portals
- The researchers cave


Scenes:
- Lost appears | Lost
- People vanish | Specter
- Portals during dreamtime


People:
- The Lost
- The Specter

cd testmy
sed -i "s/- The Specter/- The Specter\n- Wild Memories/" plan.txt
sed -i "s/- Portals during dreamtime/- Portals during dreamtime\n- Unconnected Memories/" plan.txt
hg ci -m "Added wild memories to fullfill the wish for action"
echo
hg log -G -r 19:
cd ..

@  changeset:   21:5393327d2d3f
|  tag:         tip
|  user:        my
|  date:        Sat Jan 12 00:18:20 2013 +0100
|  summary:     Added wild memories to fullfill the wish for action
|
o  changeset:   20:c00aa6f24c3f
|  user:        my
|  date:        Sat Jan 12 00:18:18 2013 +0100
|  summary:     apply people to wishes, places and scenes
|
o    changeset:   19:8bf8d55739fa
|\   parent:      17:f8cc86f681ac
| |  parent:      18:058175606243
| |  user:        my
| |  date:        Sat Jan 12 00:18:16 2013 +0100
| |  summary:     merge people
| |

cd testmy
# go back to the big change
hg up -r -2
# uncommit it
hg uncommit --all
# Now rework it into two commits
sed -i "s/- Lost appears | Lost/- Lost appears/" plan.txt
sed -i "s/- People vanish | Specter/- People vanish/" plan.txt
hg amend -m "Apply people to wishes"
sed -i "s/- Lost appears/- Lost appears | Lost/" plan.txt
sed -i "s/- People vanish/- People vanish | Specter/" plan.txt
hg commit -m "Apply people to scenes"
# let’s mark this for later use
hg book splitchanges
# and evolve to get rid of the obsoletes
echo
hg evolve --any
hg log -G -r 19:
cd ..

1 files updated, 0 files merged, 0 files removed, 0 files unresolved
new changeset is empty
(use "hg kill ." to remove it)
1 new unstable changesets

move:[21] Added wild memories to fullfill the wish for action
atop:[24] Apply people to wishes
merging plan.txt
@  changeset:   26:ab48ecaceb01
|  tag:         tip
|  parent:      24:909bb640d4fc
|  user:        my
|  date:        Sat Jan 12 00:18:20 2013 +0100
|  summary:     Added wild memories to fullfill the wish for action
|
| o  changeset:   25:76083662b263
|/   bookmark:    splitchanges
|    user:        my
|    date:        Sat Jan 12 00:18:23 2013 +0100
|    summary:     Apply people to scenes
|
o  changeset:   24:909bb640d4fc
|  parent:      19:8bf8d55739fa
|  user:        my
|  date:        Sat Jan 12 00:18:23 2013 +0100
|  summary:     Apply people to wishes
|
o    changeset:   19:8bf8d55739fa
|\   parent:      17:f8cc86f681ac
| |  parent:      18:058175606243
| |  user:        my
| |  date:        Sat Jan 12 00:18:16 2013 +0100
| |  summary:     merge people
| |

cd testmy
hg up splitchanges
hg graft -O tip
hg log -G -r 19:
cd ..

1 files updated, 0 files merged, 0 files removed, 0 files unresolved
grafting revision 26
merging plan.txt
@  changeset:   27:4d3a40c254b4
|  bookmark:    splitchanges
|  tag:         tip
|  parent:      25:76083662b263
|  user:        my
|  date:        Sat Jan 12 00:18:20 2013 +0100
|  summary:     Added wild memories to fullfill the wish for action
|
o  changeset:   25:76083662b263
|  user:        my
|  date:        Sat Jan 12 00:18:23 2013 +0100
|  summary:     Apply people to scenes
|
o  changeset:   24:909bb640d4fc
|  parent:      19:8bf8d55739fa
|  user:        my
|  date:        Sat Jan 12 00:18:23 2013 +0100
|  summary:     Apply people to wishes
|
o    changeset:   19:8bf8d55739fa
|\   parent:      17:f8cc86f681ac
| |  parent:      18:058175606243
| |  user:        my
| |  date:        Sat Jan 12 00:18:16 2013 +0100
| |  summary:     merge people
| |

Track your scientific scripts with Mercurial

If you want to publish your scientific scripts, as Nick Barnes advises in Nature, you can very easily do so with Mercurial.

All my stuff (not just code), excempting only huge datasets, is in a Mercurial source repository¹.

Whenever I change something and it does anything new, I commit the files with a simple commit (even if it’s only “it compiles!”).

With that I can always check “which were the last things I did” (look into the log) or “when did I change this line, and why?” (annotate the file). Also I can easily share my scripts folder with others and Mercurial can merge my work and theirs, so if they fix a line and I fix another line, both fixes get integrated without having to manually copy-paste them around.

For all that it doesn’t need much additional expertise: The basics can be learned in just 15 minutes — and you’ll likely never need more than these for your work².

workflow concept: automatic trusted group of committers

Goal

A workflow where the repository gets updated only from repositories whose heads got signed by at least a certain percentage or a certain number of trusted committers.

Requirements

Mercurial, two hooks for checking and three special files in the repo.

The hooks do all the work - apart from them, the repo is just a normal Mercurial repository. After cloning it, you only need to setup the hooks to activate the workflow.

Extensions: gpg

Hooks: prechangegroup and pretxnchangegroup

Files: .hgtrustedkeys , .hgbackuprepos , .hgtrustminimum

concept

Hooks

• prechangegroup: Copy the local versions of the files for access in the pretxnchangegroup hook (might be unnecessary by letting the pretxnchangegroup hook use the rollback-info).

• pretxnchangegroup:

  • per head: check if the tipmost non-signature changeset has been GnuPG signed by enough trusted keys.
  • If not all heads have enough signatures, rollback, discard the current default repo and replace it with the backup repo which has the most changesets we lack. Continue discarding bad repos until you find one with enough signatures.

Special Files

.hgtrustedkeys contains a list of public GnuPG keys.

.hgbackuprepos contains a list of (pull) links to backup repositories.

.hgtrustminimum contains the percentage or number of keys from which a signature is needed for a head to be accepted.

Notes

With this workflow you can even do automatic updates from the repository. It should be ideal for release repositories of distributed projects.

If you want to work on the project, a very worthwhile goal might be implementing it in infocalypse: anonymous code collaboration via Freenet and Mercurial, built to survive the informational apocalypse (and any kind of censorship).

For me, Gentoo is about *convenient* choice

It's often said, that Gentoo is all about choice, but that doesn't quite fit what it is for me.

After all, the highest ability to choose is Linux from scratch and I can have any amount of choice in every distribution by just going deep enough (and investing enough time).

What really distinguishes Gentoo for me is that it makes it convenient to choose.

Since we all have a limited time budget, many of us only have real freedom to choose, because we use Gentoo which makes it possible to choose with the distribution-tools. Therefore only calling it “choice” doesn't ring true in general - it misses the reason, why we can choose.

So what Gentoo gives me is not just choice, but convenient choice.

Some examples to illustrate the point:

KDE 4 without qt3

I recently rebuilt my system after deciding to switch my disk layout (away from reiserfs towards a simple ext3 with reiser4 for the portage tree). When doing so I decided to try to use a "pure" KDE 4 - that means, a KDE 4 without any remains from KDE3 or qt3.

To use kde without any qt3 applications, I just had to put "-qt3" and "-qt3support" into my useflags in /etc/make.conf and "emerge -uDN world" (and solve any arising conflicts).

Imagine doing the same with a (K)Ubuntu...

Emacs support

Similarly to enable emacs support on my GentooXO (for all programs which can have emacs support), I just had to add the "emacs" useflag and "emerge -uDN world".

Selecting which licenses to use

Just add

ACCEPT_LICENSE="-* @FSF-APPROVED @FSF-APPROVED-OTHER"

to your /etc/make.conf to make sure you only get software under licenses which are approved by the FSF.

For only free licenses (regardless of the approved state) you can use:

ACCEPT_LICENSE="-* @FREE"

All others get marked as masked by license. Default (no ACCEPT_LICENSE in /etc/make.conf) is “* -@EULA”: No unfree software. You can check your setting via emerge --info | grep ACCEPT_LICENSE. More information…

One program (suite) in testing, but the main system rock stable

Another part where choosing is made convenient in Gentoo are testing and unstable programs.

I remember my pain with a Kubuntu, where I wanted to use the most recent version of Amarok. I either had to add a dedicated Amarok-only testing repository (which I'd need for every single testing program), or I had to switch my whole system into testing. I did the latter and my graphical package manager ceased to work. Just imagine how quickly I ran back to Gentoo.

And then have a look at the ease of deciding to take one package into testing in Gentoo:

• emerge --autounmask-write =cathegory/package-version
• etc-update
• emerge =cathegory/package-version

EDIT: Once I had a note here “It would be nice to be able to just add the missing dependencies with one call”. This is now possible with --autounmask-write.

And for some special parts (like KDE 4) I can easily say something like

• ln -s /usr/portage/local/layman/kde-testing/Documentation/package.keywords/kde-4.3.keywords /etc/portage/package.keywords/kde-4.3.keywords

(I don't have the kde-testing overlay on my GentooXO, where I write this post, so the exact command might vary slightly)

Closing remarks

So to finish this post: For me, Gentoo is not only about choice. It is about convenient choice.

And that means: Gentoo gives everybody the power to choose.

I hope you enjoy it as I do!

• more about Gentoo

pmerge @glsa

#! /bin/sh


### Update the portage tree and the glsa packages via pkgcore

# spew a status message
echo $(date) "start to update GLSA" >> /tmp/cron-update.log

# Sync only portage
pmaint sync /usr/portage

# security relevant programs
pmerge -uDN @glsa > /tmp/cron-update-pkgcore-last.log || cat \
    /tmp/cron-update-pkgcore-last.log >> /tmp/cron-update.log

# And keep everything working
revdep-rebuild

# Finally update all configs which can be updated automatically
cfg-update -au

echo $(date) "finished updating GLSA" >> /tmp/cron-update.log

#!/bin/sh


### Update my computer using pgkcore, 
### since that also works if some dependencies couldn't be resolved.

# Sync all overlays
eix-sync

## First use pkgcore
# security relevant programs (with build-time dependencies (-B))
pmerge -BuD @glsa

# system, world and all the rest
pmerge -BuD @system
pmerge -BuD @world
pmerge -BuD @installed

# Then use portage for packages pkgcore misses (inlcuding overlays) 
# and for EMERGE_DEFAULT_OPTS="--keep-going" in make.conf 
emerge -uD @security
emerge -uD @system
emerge -uD @world
emerge -uD @installed

# And keep everything working
emerge @preserved-rebuild
revdep-rebuild

# Finally update all configs which can be updated automatically
cfg-update -au

alias pix='pquery --raw -nv --attr=keywords'

$ time pmerge -p @glsa
 * Resolving...
Nothing to merge.

real    0m1.863s
user    0m1.463s
sys     0m0.100s

How to make a million dollars in pay-what-you-want — thoughts on the Humble Indie Bundle

Some thoughts¹ on how the humble Indie Bundle managed to get more than 1.25 Million Dollars² in one and a half weeks — more than one quarter of that from GNU/Linux users.

Let me repeat that: One quarter of the money came from GNU/Linux users. And the average GNU/Linux user paid almost twice as much for the game as the average Windows user.

How they did it? If I could give you a simple recipe which is certain to work for everyone, I might just hire up at Blizzard.

But I think a big part is that (from my view — and obviously from the view of others, too) they did everything right. And I mean everything:

• The games are great.

• The message the name “humble indie bundle” conveys is great.

• You could pay whatever you want. From 1 cent to a million. The highest single contribution was 3,333.33$, with an average contribution of $9.17 over all platforms and $14.52 from the average GNU/Linux user³.

• You could directly see how much money they made on the front page, along with an info about the average contribution, split by platform.

• Normally each game would have cost 20$, so the average payment for all games also was a significant price drop.

• They donated about one third to charitable organizations. The buyers could decide how much should go to whom.

• Payment was easy via Paypal and others.

• All games work on GNU/Linux, MacOSX and Windows out of the box.

• Each game already had a community. The bundle bundled their impact so it went viral on Twitter, identi.ca, facebook, etc.

• They have clear and simple download links. Should I ever lose the games locally, I can just redownload them. If need be with wget.

• They use no DRM or similar, so I can show the games to friends and won’t be troubled by use restrictions.

• And on the last day they announced that for 4 of the 6 games the code would become free software if they would crack the 1 million dollar boundary. It took just over 16 more hours to raise additional 200,000$. And they followed up on their pledge with 2 games already freed and 2 more to follow as soon as the code is cleaned up.

To wrap it up: They did everything right, so almost everybody who saw it was delighted and there was nothing to break the viral network effects.

And I think that getting any one of these points wrong would have killed a major part of the network effect, because the naysayers are far stronger in the networking game than the fans.

Any foul trick would have cost them many fans, because someone would have been bound to find out and go viral with it.

Motivation and Reward

Debunking the myth of increasing the performance of creative workers with carrot and stick.

A few months ago, the GNU project had to withdraw its article on motivation and monetary reward, because its author did not allow them to spread it anymore. So I recreated the core of its message - with references to solid research.

Executive Summary

For creative tasks, the quality of performance strongly correllates with intrinsic motivation: Being interested in the task itself.

This article will only talk about that.

The main factors which are commonly associated with intrinsic motivation are:

• Positive verbal feedback which increases intrinsic motivation.
• Payment independent of performance which actually has no effect.
• Payment dependent on performance which reduces the motivation on the long term.
• Negative verbal feedback which directly reduces intrinsic motivation.
• Threatening someone with punishment which strongly reduces intrinsic motivation.

To make it short: Anything which diverts the focus from the task at hand towards some external matter (either positive or negative) reduces the intrinsic motivation and that in turn reduces work performance.

If you want to help people perform well, make sure that they don’t have to worry about other stuff besides their work and give them positive verbal feedback about the work they do.

Background

Since this claim goes pretty much against the standard ideology of market-trusting economists, I want to back it with solid scientific background.

The easiest way to do that is going to google scholar and searching for research on motivation and rewards. It gives a meta-analysis of experiments on the effects of extrinsic rewards on intrinsic motivation:

A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation.
— E.L. Deci, R Koestner, R.M. Ryan - Psychological bulletin, 1999 - psycnet.apa.org

This paper is cited by 2324 other papers Google knows about, which is an indicator of being accepted by the psychological community (except if it should have 2324 rebuttals) - an indicator which even those can understand who are not really versed in that community (for example me).

I dug into the paper to find solid scientific research on the effects of payment on motivation. And that led me to this older paper from Edward L. Deci:

The Effects of Contingent and Noncontingent Rewards and Controls on Intrinsic Motivation
— Edward L. Deci, University of Rochester, Organizational Behavior and Human Performance, 1972

Their research question was trying to find out if money paid unconditionally weakens intrinsic motivation like money paid for good performance:

» Two recent papers (Deci, 1971, 1972) have presented evidence that when money was paid to subjects for performing intrinsically motivated activities, and when that money was made contingent on their performance, they were less intrinsically motivated after the experience with money than were subjects who performed the same activity for no pay.«

This is about intrinsic motivation: The kind of motivation which fuels artists and other creative people and allows them to do great deeds.

It’s the kind of motivation, a company should try to inspire in every employee who does anything remotely creative or complex.

What reduces intrinsic motivation

There was previous research which showed a reduction of intrinsic motivation due to payment. To make their research solid, the first thing E.L. Deci and his group did was a replication to ensure that the basic theory is correct.

In another experiment using the one-session paradigm, Deci and Cascio (1972) showed that negative feedback resulting from bad performance on an intrinsically motivated activity caused a decrease in intrinsic motivation.

In my words: Tell people that they do bad work and you reduce their motivation - not surprisingly.

“Your performance sucks” → intrinsic motivation decreases.

Further, Deci and Caseio (1972) reported that when subjects were threatened with punishment for poor performance, their intrinsic motivation also decreased.

Threaten people, and their motivation gets reduced, too.

“If you fail, you’re fired” → intrinsic motivation decreases.

[…]Deci (1972) replicated the finding that subjects who were paid one dollar per puzzle solved showed a decrease in intrinsic motivation.

Pay people for good performance and you reduce their motivation.

“For each housing loan you sell, you get 20€” → intrinsic motivation decreases.

This is the result which actually marks all the performance-based payment schemes which are so popular with the administration folks as utter nonsense - at least for creative and complex jobs.

For those jobs your employees enjoy doing, bonusses actually decrease performance on the long run. These are the kinds of jobs in which people can work overnight and concentrated for hours and lose track of time while they work on systems which are too complex for most people to even pretend to understand. The kind of jobs where some people get into the flow and do more work in an hour than other people do in a week. Jobs in science, in programming and actually in any other topic in which you do not just follow prescribed rules but actually solve problems.

The kind of jobs which is more and more common, because jobs with prescribed rules can just as well be done by machines.

And social jobs, the other kind of jobs for which you need people, because people doing social jobs work with people and anything involving people is a complex problem by definition. At least if you want really good results.

Or, seen from a different perspective: If two companies compete in a segment of the market and one has motivated people and the other doesn’t - and other factors are mostly equal - then the company with motivated people wins.

So you want motivated people. And in creative, complex or social jobs, you want them intrinsically motivated. You want them to do a good job for the sake of doing a good job.

Which means, you want to avoid

• giving them negative feedback,
• threatening them and
• paying them based on their performance.

With that in mind, let us go on: How can we actually motivate people?

What enhances motivation

To answer that, let’s listen to research again:

On the other hand, Deei (1971, 1972) has reported that verbal reinforcements do not decrease intrinsic motivation; in fact, they appear to enhance it.

So, to increase motivation, tell people that they do good work.

„I like that plan! Go for it!“ → intrinsic motivation increases.

That’s all you can do. Tell them that they do good work. Encourage them.

But isn’t there a paradox? How can we actually employ people, if paying them money for good work decreases their motivation?

How to pay motivated people?

That’s the real question, the paper from Edward L. Deci tackled:

While extrinsic rewards such as money can certainly motivate behavior, they appear to be doing so at the expense of intrinsic motivation. […but…] when payments were not contingent upon performance, intrinsic motivation did not decrease.

So the answer is pretty simple: Just pay them money independent of how well they do.

„You get 3000€ a month. Flat. That’s enough to lead a good life.“¹ → intrinsic motivation stays stable.

The real trick is to just give them money, independent of how well they do. If motivated people work for you, ensure that they do not have to worry about money. Do all you can to take money concerns off their mind.

And tell them what they do well.

At least that’s what you should do if you want to base your actions on research instead of on the broken intuition of people who get paid for their performance in convincing you of their ideology (and consequently often do so in blatant, uncreative ways).

If you do that already: That’s great! Likely it’s really cool to work with you.

Illustration

A very illustrative experiment on losing intrinsic interest due to external reward was done by Lepper, Mark R.; Greene, David; Nisbett, Richard E..²

They observed three groups of pre-school children. The first group was told that they would get a “certificate with a gold seal and ribbon” if they would draw something. The second group wasn’t told that they would get a reward, but got it after drawing, too. The third group did not get any reward and did not expect any.

Before the start of the experiment, their intrinsic interest in drawing was measured by observing how much time they spent drawing when they had the chance.

One to two weeks after the experiment, the intrinsic interest of the children was measured again by observing them through a one-way mirror.

In that subsequent measurement, the children who had been told that they would get the reward for drawing (and had gotten the reward) used half as much time for drawing as those who had not gotten any reward or those who had gotten an unexpected reward.

And even when the pictures which they had drawn during the initial test were compared, the pictures from the group who expected a reward were of significantly lower quality than the pictures from the two other groups. the difference between expected extrinsic reward and no reward was 2.18 vs. 2.69 on an independently judged quality scale between 1 (very poor) and 5 (very good).

So offering children a reward for drawing not only reduces their intrinsic interest in drawing, but also reduces the quality of the pictures they draw.

And this is perfectly in line with the results from the paper from Edward L. Deci on intrinsic motivation of adults.

Summary

To increase the motivation of people, DO

• Pay them a good monthly income, so they don’t have to worry about money, and
• Give them positive verbal feedback on the things they do well.

And should you happen to be interested in helping a free software project with money, just employ some of the people hacking on the project - and give them a good, longterm contract with enough freedom of choice, so they don’t have to worry about money or what they are allowed to do, but can instead focus on working to make the project succeed - like they did before you employed them, but now with much more time at their disposal. And, as with anything else, give them positive feedback on the things they do well.

If you want to help people perform well, make sure that they don’t have to worry about other stuff besides their work and give them positive verbal feedback about the work they do.

No, it ain’t “forever” (GNU Hurd code_swarm from 1991 to 2010)

If the video doesn’t show, you can also download it as Ogg Theora & Vorbis “.ogv” or find it on youtube.

This video shows the activity of the Hurd coders and answers some common questions about the Hurd, including “How stagnated is Hurd compared to Duke Nukem Forever?”. It is created directly from commits to Hurd repositories, processed by community codeswarm.

Every shimmering dot is a change to a file. These dots align around the coder who did the change. The questions and answers are quotes from todays IRC discussions (2010-07-13) in #hurd at irc.freenode.net.

You can clearly see the influx of developers in 2003/2004 and then again a strenthening of the development in 2008 with less participants but higher activity than 2003 (though a part of that change likely comes from the switch to git with generally more but smaller commits).

I hope you enjoyed the high-level look on the activity of the Hurd project!

PS: The last part is only the information title with music to honor Sean Wright for allowing everyone to use and adapt his Album Enchanted.

Some technical advantages of the Hurd

→ An answer to just accept it, truth hurds, where Flameeyes told his reasons for not liking the Hurd and asked for technical advantages (and claimed, that the Hurd does not offer a concept which got incorporated into other free software, contributing to other projects). Note: These are the points I see. Very likely there are more technical advantages which I don’t see well enough to explain them. Please feel free to point them out.

Information for potential testers: The Hurd is already usable, but it is not yet in production state. It progressed a lot during the recent years, though. Have a look at the status report if you want to see if it’s already interesting for you.

Thanks for explaining your reasons. As answer:

Firstoff: FUSE is essentially an implementation of parts of the translator system (which is the main building block of the Hurd) to Linux, and NetBSD recently got a port of the translators system of the Hurd. That’s the main contribution to other projects that I see.

On the bare technical side, the translator-based filesystem stands out: The filesystem allows for making arbitrary programs responsible for displaying a given node (which can also be a directory tree) and to start these programs on demand. To make them persistent over reboots, you only need to add them to the filesystem node (for which you need the right to change that node). Also you can start translators on any node without having to change the node itself, but then they are not persistent and only affect your view of the filesystem without affecting other users. These translators are called active, and you don’t need write permissions on a node to add them.

The filesystem implements stuff like Gnome VFS (gvfs) and KDE network transparency on the filesystem level, so those are available for all programs. And you can add a new filesystem as simple user, just as if you’d just write into a file “instead of this node, show the filesystem you get by interpreting file X with filesystem Y” (this is what you actually do when setting a translator but not yet starting it (passive translator)).

One practical advantage of this is that the following works:

settrans -a ftp\: /hurd/hostmux /hurd/ftpfs /
dpkg -i ftp://ftp.gnu.org/path/to/*.deb

This installs all deb-packages in the folder path/to on the FTP server. The shell sees normal directories (beginning with the directory “ftp:”), so shell expressions just work.

You could even define a Gentoo mirror translator (settrans mirror\: /hurd/gentoo-mirror), so every program could just access mirror://gentoo/portage-2.2.0_alpha31.tar.bz2 and get the data from a mirror automatically: wget mirror://gentoo/portage-2.2.0_alpha31.tar.bz2

Or you could add a unionmount translator to root which makes writes happen at another place. Every user is able to make a readonly system readwrite by just specifying where the writes should go. But the writes only affect his view of the filesystem.

Starting a network process is done by a translator, too: The first time something accesses the network card, the network translator starts up and actually provides the device. This replaces most initscripts in the Hurd: Just add a translator to a node, and the service will persist over restarts.

It’s a surprisingly simple concept, which reduces the complexity of many basic tasks needed for desktop systems.

And at its most basic level, Hurd is a set of protocols for messages which allow using the filesystem to coordinate and connect processes (along with helper libraries to make that easy).

Also it adds POSIX compatibility to Mach (while still providing access to the capabilities-based access rights underneath, if you need them). You can give a process permissions at runtime and take them away at will. For example you can start all programs without permission to use the network (or write to any file) and add the permissions when you need them.

groups # → root
addauth -p $(ps -L) -g mail
groups # → root mail 

And then there are subhurds (essentially lightweight virtualization which allows cutting off processes from other processes without the overhead of creating a virtual machine for each process). But that’s an entire post of its own…

And the fact that a translator is just a simple standalone program means that these can be shared and tested much more easily, opening up completely new options for lowlevel hacking, because it massively lowers the barrier of entry.

And then there is the possibility of subdividing memory management and using different microkernels (by porting the Hurd layer, as partly done in the NetBSD port), but that is purely academic right now (search for Viengoos to see what its about).

So in short: The translator system in the Hurd is a simple concept which makes many tasks easy, which are complex with Linux (like init, network transparency, new filesystems, …). Additionally there are capabilities, subhurds and (academic) memory management.

Best wishes,
Arne

PS: I decided to read flameeyes’ post as “please give me technical reasons to dispell my emotional impression”.

PPS: If you liked this post, it would be cool if you’d flattr it:

PPPS: Additional information can be found in Gaël Le Mignot’s talk notes, in niches for the Hurd and the GNU Hurd documentation pages.

P4S: This post is also available in the Hurd Staging Wiki.

(A)GPL as hack on a Python-powered copyright system

AGPL is a hack on copyright, so it has to use copyright, else it would not compile/run.

All the GPL licenses are a hack on copyright. They insert a piece of legal code into copyright law to force it to turn around on itself.

You run that on the copyright system, and it gives you code which can’t be made unfree.

To be able to do that, it has to be written in copyright language (else it could not be interpreted).

my_code = "<your code>"

def AGPL ( code ): 
    """
    >>> is_free ( AGPL ( code ) )
    True
    """
    return eval (
        transform_to_free ( code ) )

copyright ( AGPL ( my_code ) )

You pass “AGPL ( code )” to the copyright system, and it ensures the freedom of the code.

The transformation means that I am allowed to change your code, as long as I keep the transformation, because copyright law sees only the version transformed by AGPL, and that stays valid.

Naturally both AGPL definition and the code transformed to free © must be ©-compatible. And that means: All rights reserved. Else I could go in and say: I just redefine AGPL and make your code unfree without ever touching the code itself (which is initially owned by you by the laws of ©):

def AGPL ( code ): 
    """ 
    >>> is_free ( AGPL ( code ) )
    False
    """
    return eval (
        transform_to_mine ( code ) )

In this Python-powered copyright-system, I could just define this after your definition but before your call to copyright(), and all calls to APGL ( code ) would suddenly return code owned by me.

Or you would have to include another way of defining which exact AGPL you mean. Something like “AGPL, but only the versions with the sha1 hashes AAAA BBBB and AABA”. cc tries to use links for that, but what do you do if someone changes the DNS resolution to point creativecommons.org to allmine.com? Whose DNS server is right, then - legally speaking?

In short: AGPL is a hack on copyright, so it has to use copyright, else it would not compile/run.

7,26€ through Flattr last month

Last month I earned 7,26€ through my Flattr account (Flattr is a voluntary payment service where people can make micropayments if they like something - after enjoying it). The flattrs came in through just 4 items:

• The incremental Mercurial guide, “Learning Mercurial in Workflows”.
• The Freenet promotion “I hope French Filesharers turn to Freenet” (written in reply to news about millions of hadopi letters).
• The Esperanto version of my free one page pen-and-paper roleplaying game: The “Slipoj-Rolludo”.
• And my free software song “Infinite Hands”.

Thank you very much for your flattrs, dear supporters¹! Thanks to you I could pay most of my server cost this month via the money from flattr - and that’s great!²

A simple solution to the dining philosophers problem

The problem

5 Philosophers do nothing but eat and think.

They have a table with 5 chairs, 5 plates and 5 forks.

Each of them eats with two forks.

Ensure that none of them starves.

The solution

First I teach them to always take the left fork first.

Then I smash one of their chairs.

Explanation

Since they can't repair the chair (they think, but they don't build), there are only 4 places left, and so they have one leftover fork which gets passed on, once one finished eating.

Inspired by Willim Stallings' Operating systems: "Use a servant who lets only 4 dine at the same time"

Naturally now they have to either change places or move chairs, so they might still need a servant :)

Censorship in the Streets — it’s idiocy everywhere

A man in the streets faces a knife.
Two policemen are there it once. They raise a sign:

“Illegal Scene! Noone may watch this!”

The man gets robbed and stabbed and bleeds to death.
The police had to hold the sign.

…

Welcome to Europe, citizen. Censorship is beautiful.

→ Courtesy to Censilia, who wants censorship in the EU after it failed in Germany. You might also be interested in 11 more reasons why censorship is useless and harmful.

PS: This poem is free licensed. Please feel free to use it anyway you like, as long as you provide a backlink.

def censor_the_net_2012()

def censor_the_net():
  try: SOPA() # see Stop Online Piracy Act 
  except Protest: # see sopastrike.com 
    try: PIPA() # see PROTECT IP Act 
    except Protest: # see weak links 
      try: OPEN() # see red herring 
      except Protest: 
        try: ACTA() # see Anti-Counterfeiting_Trade_Agreement 
        except Protest: # see resignation⁽¹⁾, court, vote anyway and advise against
          try: CISPA() # see Stop the Online Spying Bill 
          except Protest: # see Dangers
            do_it_anyway() # destroy free speech and computers (english video).

while wealth_breeds_wealth and wealth_gives_power: # (german text and english video)
  censor_the_net() # see wealth vs. democracy (german)

This code is valid Python.

Feel free to use and change this snippet, as long as you include a reference to this page (http://draketo.de/node/475 or http://draketo.de/light/english/politics/def-censor-the-net-2012) or my name (Arne Babenhauserheide).

Here’s the linked english video, embedded (external, not GPL!):

Going from a simple Makefile to Autotools

Table of Contents

1 Intro

I recently started looking into Autotools, to make it easier to run my code on multiple platforms.

Naturally you can use cmake or scons or waf or ninja or tup, all of which are interesting in there own respect. But none of them has seen the amount of testing which went into autotools, and none of them have the mount of tweaks needed to support about every system under the sun. And I recently found pyconfigure which allows using autotools with python and offers detection of library features.

I had already used Makefiles for easily storing the build information of anything from python projects (python setup.py build) to my PhD with all the required graphs.

I also had used scons for those same tasks.

But I wanted to test, what autotools have to offer. And I found no simple guide which showed me how to migrate from a Makefile to autotools - and what I could gain through that.

So I decided to write it.

all: doktorarbeit.pdf sink.pdf

sink.pdf : sink.tex images/comp-t3-s07-tem-boas.png images/comp-t3-s07-tem-bona.png images/bona-marble.png images/boas-marble.png
    pdflatex sink.tex
    rm -f  *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb # kill litter

comp-t3-s07-tem-boas.png comp-t3-s07-tem-bona.png : nee-comp.pyx nee-comp.txt
    pyxplot nee-comp.pyx

doktorarbeit.pdf : doktorarbeit.org
    emacs --batch --visit "doktorarbeit.org" --funcall org-export-as-pdf  

# comments start with #
thing : required source files # separated by spaces
    build command with the files
# ^ this is a TAB.

dnl run `autoreconf -i` to generate a configure script. 
dnl Then run ./configure to generate a Makefile.
dnl Finally run make to generate the project.

AC_INIT([Doktorarbeit Inverse GHG], [0.1], [arne.babenhauserheide@kit.edu])
dnl we use the build type foreign here instead of gnu because I do not have a NEWS file and similar, yet.
AM_INIT_AUTOMAKE([foreign])
AC_CONFIG_FILES([Makefile])
AC_OUTPUT

pdf_DATA = sink.pdf doktorarbeit.pdf

sink = sink.tex
pkgdata_DATA = images/comp-t3-s07-tem-boas.png images/comp-t3-s07-tem-bona.png
dist_pkgdata_DATA = images/bona-marble.png images/boas-marble.png

plotdir = .
dist_plot_DATA = nee-comp.pyx nee-comp.txt

doktorarbeit = doktorarbeit.org

EXTRA_DIST = ${sink} ${dist_pkgdata_DATA} ${doktorarbeit}

MOSTLYCLEANFILES = \#* *~ *.bak # kill editor backups
CLEANFILES = ${pdf_DATA}
DISTCLEANFILES = ${pkgdata_DATA}

sink.pdf : ${sink} ${pkgdata_DATA} ${dist_pkgdata_DATA}
    TEXINPUTS=${TEXINPUTS}:$(srcdir)/:$(srcdir)/images// pdflatex $<
    rm -f  *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb # kill litter

${pkgdata_DATA} : ${dist_plot_DATA}
    $(foreach i,$^,if test "$(i)" != "$(notdir $(i))"; then cp -u "$(i)" "$(notdir $(i))"; fi;)
    ${MKDIR_P} images
    pyxplot $<
    $(foreach i,$^,if test "$(i)" != "$(notdir $(i))"; then rm -f "$(notdir $(i))"; fi;)

doktorarbeit.pdf : ${doktorarbeit}
    if test "$<" != "$(notdir $<)"; then cp -u "$<" "$(notdir $<)"; fi
    emacs --batch --visit "$(notdir $<)" --funcall org-export-as-pdf
    if test "$<" != "$(notdir $<)"; then rm -f "$(notdir $<)"; rm -f $(basename $(notdir $<)).tex $(basename $(notdir $<)).tex~; else rm -f $(basename $<).tex $(basename $<).tex~; fi

dnl run `autoreconf -i` to generate a configure script. 
dnl Then run ./configure to generate a Makefile.
dnl Finally run make to generate the project.

AC_INIT([Doktorarbeit Inverse GHG], [0.1], [arne.babenhauserheide@kit.edu])
# Check for programs I need for my build
AC_CANONICAL_TARGET
AC_ARG_VAR([emacs], [How to call Emacs.])
AC_CHECK_TARGET_TOOL([emacs], [emacs], [no])
AC_ARG_VAR([pyxplot], [How to call the Pyxplot plotting tool.])
AC_CHECK_TARGET_TOOL([pyxplot], [pyxplot], [no])
AC_ARG_VAR([pdflatex], [How to call pdflatex.])
AC_CHECK_TARGET_TOOL([pdflatex], [pdflatex], [no])
AS_IF([test "x$pdflatex" = "xno"], [AC_MSG_ERROR([cannot find pdflatex.])])
AS_IF([test "x$emacs" = "xno"], [AC_MSG_ERROR([cannot find Emacs.])])
AS_IF([test "x$pyxplot" = "xno"], [AC_MSG_ERROR([cannot find pyxplot.])])
# Run automake
AM_INIT_AUTOMAKE([foreign])
AM_MAINTAINER_MODE([enable])
AC_CONFIG_FILES([Makefile])
AC_OUTPUT

pdf_DATA = sink.pdf doktorarbeit.pdf

sink = sink.tex
pkgdata_DATA = images/comp-t3-s07-tem-boas.png images/comp-t3-s07-tem-bona.png
dist_pkgdata_DATA = images/bona-marble.png images/boas-marble.png

plotdir = .
dist_plot_DATA = nee-comp.pyx nee-comp.txt

doktorarbeit = doktorarbeit.org

EXTRA_DIST = ${sink} ${dist_pkgdata_DATA} ${doktorarbeit}

MOSTLYCLEANFILES = \#* *~ *.bak # kill editor backups
CLEANFILES = ${pdf_DATA}
DISTCLEANFILES = ${pkgdata_DATA}

sink.pdf : ${sink} ${pkgdata_DATA} ${dist_pkgdata_DATA}
    TEXINPUTS=${TEXINPUTS}:$(srcdir)/:$(srcdir)/images// @pdflatex@ $<
    rm -f  *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb # kill litter

${pkgdata_DATA} : ${dist_plot_DATA}
    $(foreach i,$^,if test "$(i)" != "$(notdir $(i))"; then cp -u "$(i)" "$(notdir $(i))"; fi;)
    ${MKDIR_P} images
    @pyxplot@ $<
    $(foreach i,$^,if test "$(i)" != "$(notdir $(i))"; then rm -f "$(notdir $(i))"; fi;)

doktorarbeit.pdf : ${doktorarbeit}
    if test "$<" != "$(notdir $<)"; then cp -u "$<" "$(notdir $<)"; fi
    @emacs@ --batch --visit "$(notdir $<)" --funcall org-export-as-pdf
    if test "$<" != "$(notdir $<)"; then rm -f "$(notdir $<)"; rm -f $(basename $(notdir $<)).tex $(basename $(notdir $<)).tex~; else rm -f $(basename $<).tex $(basename $<).tex~; fi  

#!/usr/bin/env python
## I need a couple of special builders for my projects
# the $SOURCE replacement only uses the first source file. $SOURCES gives all.
# specifying all source files makes it possible to rerun the build if a single source file changed.
orgexportpdf = 'emacs --batch --visit "$SOURCE" --funcall org-export-as-pdf'
pyxplot = 'pyxplot $SOURCE'
# pdflatex is quite dirty. I directly clean up after it with rm.
pdflatex = 'pdflatex $SOURCE -o $TARGET; rm -f  *_flymake* flymake* *.log *.out *.toc *.aux *.snm *.nav *.vrb'

# build the PhD thesis from emacs org-mode.
Command("doktorarbeit.pdf", "doktorarbeit.org",
        orgexportpdf)

# create plots
Command(["images/comp-t3-s07-tem-boas.png", 
         "images/comp-t3-s07-tem-bona.png"], 
        ["nee-comp.pyx", 
         "nee-comp.txt"],
        pyxplot)

# build my sink.pdf
Command("sink.pdf", 
        ["sink.tex", 
         "images/comp-t3-s07-tem-boas.png", 
         "images/comp-t3-s07-tem-bona.png", 
         "images/bona-marble.png", 
         "images/boas-marble.png"],
        pdflatex)

# My editors leave tempfiles around. I want them gone after a build clean. This is not yet supported!
tempfiles = Glob('*~') + Glob('#*#') + Glob('*.bak')
# using this here would run the cleaning on every run.
#Command("clean", [], Delete(tempfiles))

sconsrun : scons
    python scons/bootstrap.py -Q

scons : 
    hg clone https://bitbucket.org/ArneBab/scons

#+BEGIN_LaTeX
%%% Local Variables:
%%% TeX-master: t
%%% End:
#+END_LaTeX

How to make companies act ethically

→ comment on Slashdot concerning Unexpected methods to promote freedom?

Was it really Apple who ended DRM? Would they have done so without the protests and evangelizing against DRM? Without protesters in front of Apple Stores? And without the many people telling their friends to just not accept DRM?

That “preaching” created a situation where Apple could reap monetary gain from doing the right thing. You see how they act when the stakes are diffecent.

What you can do to make companies act ethically is to create a situation where they can make more money by working ethically than by ripping you off. The ways to do that are

1. Laws (breaking them costs money when you get caught),
2. Taxes on doing the wrong thing (i.e. pollution),
3. Offering your work in ways, which make it easier for people to make money ethically than unethically (that’s what copyleft licensing does),
4. Trying to convince people to do 3,
5. Trying to convince people to shun products which are created unethically (that’s what you call preaching),
6. Only paying for products which were produced ethically.

RMS does 3,4, 5 and 6, so he’s pretty much into gaming the market - and “preaching” is only one tool in his box. Though what he does is more convincing than preaching: He gives us reasons why unfree software is bad - and the mental tools to resist the preaching from the other side (for example via analyses of speech-tricks, like calling state-granted monopolies “property”).

identi.ca Group: Out of Group (!oog)

What !oog is

The Out of Group group is a way to request leading an overboarding discussion out of group (so you don't spam all the people who are in the group where the discussion started, but who simply want news).

Motto

Please discuss out of group. You can wrap up the discussion afterwards (link to the context) and add a group tag then.

How To

To request taking a discussion out of group, simply join !oog, add !oog to your message and then leave the group again (except if you want to see other !oog requests).

For example you can use the following to request moving !oog:

Please let us continue the discussion !oog and wrap it up afterwards. It disturbs others in here. !group1 !group2

Background

This is a reaction to a discussion about the use of group-tags in discussions.

Archived discussion

Available under the Creative Commons Attribution 3.0 license.

• rysiek
  @teddks: stop using group tags, please. everybody on !ubuntu and !linux have heard enough, really.

• teddks
  @rysiek I respond to in-group messages in-group. If you don't want me to use them, don't use them to me. !ubuntu !linux

• arnebab
  @rysiek @teddks please leave the group tags out, both of you.

• teddks
  @arnebab My policy for group-posting rebuttals was posted a bit ago.

• arnebab
  @teddks please discuss out of the group. You can wrap up the discussion afterwards and add a group tag then. You're one post from a block.

• arnebab
  @teddks By not discussing in the group you make the others' in-group posts look silly. Otherwise you just look silly yourself.

• teddks
  @arnebab Wait, what? I only discuss in-group if the previous post was in-group.

• arnebab
  @teddks That principle doesn't scale. If we all used it groups would be useless. A wrapup post can get people to read all http://is.gd/6CoYI

• teddks
  @arnebab I might start doing that for oog discussions, but I'm not going to deny myself the same forum my opponents have.

• arnebab
  @teddks they are not opponents but discussion partners. And they look silly if they stay ingroup while you post oog. Just ask them to go oog

• arnebab
  @teddks and wrap it up later. If they insist on staying ingroup, just post one ingroup request to come oog and let peer pressure do the rest

• arnebab
  @teddks they broadcast to people who aren't interested and will block them.

• arnebab
  @teddks sorry for the phraselike answer. 140 chars aren't ideal for discussing more complex topics...

• teddks
  @arnebab I understand. I kind of regret how identi.ca has taken the place of IRC for a lot of things.

• teddks
  @arnebab That's irrelevant; in-group they get to broadcast their arguments and their views. I'm not going to deny myself that.

• arnebab
  @teddks That means I have to block you when you post your next ingroup broadcast. You lose all readers that way.

• teddks
  @arnebab Now you are inviting spam and personal attacks against me in !ubuntu. Is that pursuant to the Code of Conduct?

• arnebab
  @teddks You do know that people can see the context with one click, do you?

• teddks
  @arnebab I don't see what you're implying - that I should depend on clickthroughs to have my arguments be heard?

• arnebab
  @teddks Since you kept spamming the !ubuntu and !linux group and explicitely said you won't stop ( http://is.gd/6Cucx ) I blocked you.

• teddks
  @arnebab I can't respond fully now, but I will later. I'm sorry that the #Ubuntu group's de facto policy is now one of censorship.

Note: arnebab was no member of the ubuntu group at that time. The block was/is a purely personal one: Nothing teddks writes will appear on arnebabs timeline, till he unblocks teddks.

Internet, community cloud foo and control of my own data

Why?

What I miss in the internet is the notion of being able to control what my apps access for data.

Why can’t a chat application just connect to a neighborhood- or community-server, and why can’t the activity-stream come from the people I know — and query only their systems, like jabber does?

Almost all geolocation services should be implementable over direct friend-to-friend connections like jabber, and I don’t really see why my local identi.ca program can’t also get the news from my local jabber contacts.

Or why I can’t set a local info-provider as geolocation source and have a “phone-book” of info-providers in each town.

And when it can do that, why can’t I have a general info-server which serves as synchronization and aggregation service for any of my devices, so all my programs on any device know which sources to use?

And why can’t I tell that server to allow my friends to access a subset of my data — selected by me?

Sadly I assume that the answer is “power”. Google and Apple don’t want to lose their control on synchronization and sharing. Otherwise most of the control and centralization (=moneymaking monopoly) of the internet would fade away.

What?

For example I’d like to be able to select whose information I get, and I’d like to be able to also get the information my friends and their get. Without anyone outside knowing that I access that data (because I ask them directly). And ideally also without me knowing from which of their friends the data originates, but still being able to block those individually.

Then I could allow certain product information providers (=good advertisers) inside my network, so I get news about stuff I might like to spend money on. And automatically get information about the info-providers from my friends — or my community.

And all that without direct dependency on a single company or system.

It would make it infeasible to monopolize the services without making everyone trust you — and having to make sure most people trust you creates a reverse-dependency which could help to keep the information-providers honest.

How?

And I think one key to that is to make that service less like a full-storage and more like update-collecting and synchronization services.

There’s no reason why a synchro-server should keep any data I already pulled to all my devices.

This would be similar to using a Mercurial push-cache of kinds: When I push data to a service, it just stores a bundle against the revision of the data on my least up-to-date device. All my devices can access that bundle, and when all are up to at least a certain state, the now useless data gets stripped out and only the new data remains.

Not yet pulled information could be stored as snapshots, until the first of my devices pulls it. Then it could get replaced by synchronization data — a compressed update-bundle. That would also make sure that incoming data has to be integrated and parsed only once.

→ http://mercurial.selenic.com/

…

Maybe Akonadi (from KDE) can someday accomplish something like that.

→ http://userbase.kde.org/Glossary#Akonadi
→ http://en.wikipedia.org/wiki/Akonadi

PS: Originally this started as a comment to The state of the internet operating system by O’Reilly.

Neither Humble nor Indie Bundle

Comment to New Humble Bundle Is Windows Only, DRM Games.

The new Humble Indie Bundle is no longer free, indie, cross-plattform or user-respecting.

When the first bundle had a huge boost in last-minute sales after the devs offered to free the source of 4 of the 5 games, I had hoped, they would keep that. I was one of those who paid when they offered to free the games, and I’m pretty sure that they got a huge boost in people who knew the Humble Indie Bundle due to that.

But when the second bundle did not offer freeing the source, I did not pay. Unfree games aren’t worth much to me and I feared they would go further down that track.

Now Steam comes to GNU/Linux, so being cross-plasform isn’t unique for the Humble Indie Bundle anymore. And they dropped cross-platform support and added DRM. They replaced fans with short-term cash-cows who will happily switch to another project without second thoughts. Somehow I saw that coming…

Well, they sell their brand while it still holds, but by doing that they burn the ones who brought them where they are today.

Never put effort in a project where you have to trust the creator to not misuse it. Free copyleft licenses are a safeguard for contributors - not only the coders, but also for those who promote the project.¹

Ogg Theora and h.264 - which video codec as standard for internet-video?

We had a kinda long discussion on identi.ca about Ogg Theora and h.264, and since we lacked a simple comparision method, I hacked up a quick script to test them.

It uses frames from Big Buck Bunny and outputs the files bbb.ogg and bbb.264 (license: cc by).

The ogg file looks like this:

The h.264 file looks like this: download

Results

What you can see by comparing both is that h.264 wins in terms of raw image quality at the same bitrate (single pass).

So why am I still strongly in favor of Ogg Theora?

The reason is simple:

Due to licensing costs of h.264 (a few millions per year, due from 2015 onwards) making h.264 the standard for internet video would have the effect that only big companies would be able to make a video enabled browser - or we would get a kind of video tax for free software: if you want to view internet video with free software, you have to pay for the right to use the x264 library (else the developers couldn't cough up the money to pay for the parent license). And noone but the main developers and huge corporations could distribute the x264 library, because they’d have to pay license fees for that.

And noone could hack on the browser or library and distribute the changed version, so the whole idea of free software would be led ad absurdum. It wouldn't matter that all code would be free licensed, since only those with a h.264 patent license could change it.

So this post boils down to a simple message:

“Support !theora against h.264 and #flash [as video codec for the web]. Otherwise only big companies will be able to write video browsers - or we get a h.264 tax on !fs”

Theoras raw quality may still be worse, but the license costs and their implications provide very clear reasons for supporting Theora - which in my view are far more important than raw technical stuff.

The test-script

for k in {0..1}
     do for i in {0..9}
         do for j in {0..9}
             do
wget http://media.xiph.org/BBB/BBB-360-png/big_buck_bunny_00$k$i$j.png
         done
     done
done

mplayer -vo yuv4mpeg -ao null -nosound mf://*png -mf fps=50

theora_encoder_example -z 0 --soft-target -V 400 -o bbb.ogg stream.yuv

mencoder stream.yuv -ovc x264 -of rawvideo -o bbb.264 -x264encopts bitrate=400 -aspect 16:9 -nosound -vf scale=640:360,harddup

p2p-networks help law enforcement catch hard criminals

Comment to: Local man faces court on child pornography charges by heraldstandard.com

As I see it, the only way the authorities did track him was due to his use of p2p-networks.

At the moment, technology makes it relatively easy for the police to track hard criminals in p2p-networks, but it also allows people to do small infringements rather safely (just like people don't stop at red traffic lights when there is no car in sight),

So I'd think the current state quite ideal.

Sadly there's an organisation which drives p2p-networks underground and which will eventually cease that action or archieve the "fame" to have been the one organisation which was responsible in the end for forcing p2p-networks to evolve into completely anonymous and untrackeable networks, where hard crimes aren't trackeable anymore.

So, this case shows once again, that "piracy" shouldn't be attacked but should instead be allowed and even fostered, because they increase social welfare (the access to media is improved, while there is no significant damage to sales) and in many cases even helped law enforcement catch ciminals who really do damage (and in this case: did very much damage).

Information about the impact of p2p-networks based on a study from the university of chicago:
- http://www.journals.uchicago.edu/JPE/journal/issues/v115n1/31618/31618.h...
- http://www.journals.uchicago.edu/cgi-bin/resolve?JPE31618PDF
(open twice to read)

Patent law overrides copyright breaks ownership

Concise and clear.

In patent law, copyright and property there are two pillars: protection and control.

Protection

• Property: No person shall take that from me.
• Copyright: No person shall have the same without my permission. A monopoly.
• Patent Law: No person shall create something similar without my permission. An even stronger monopoly.

Control

• Property: I decide what happens with this.
• Copyright: I decide what happens to everything which is the same. Takes another ones property. → a monopoly¹.
• Patent Law: I decide what happens to every similar thing. Takes the copyright and property of others. → An even stronger monopoly¹.

In short: Patent law overrides copyright breaks ownership.

¹: Others may have copyrights and property rights which they can only exercise with my permission. So effectively all their rights belong to me. If you want a longer argument on this, please read Intellectual Property Is Theft.

(translation of Patentrecht bricht Urheberrecht bricht Eigentum)

Phoronix conclusions distort their results, shown with the example of GCC vs. LLVM/Clang On AMD's FX-8350 Vishera

Phoronix recently did a benchmark of GCC vs. LLVM on AMD hardware. Sadly their conclusion did not fit the data they showed. Actually it misrepresented the data so strongly, that I decided to speak up here instead of having my comments disappear in their forums.

Taking out the OpenMP benchmarks (where GCC naturally won, because LLVM only processes those tests single-threaded) and the build times (which are irrelevant to the speed of the produced binaries), their benchmark had the following result:

LLVM is slower than GCC by:

• 10.2% (HMMer)
• 12.7% (MAFFT)
• 6.8% (BLAKE2)
• 9.1% (HIMENO)
• 42.2% (C-Ray)

With these results (which were clearly visible on their result summary on OpenBenchmarking, Michael Larabel from Phoronix concluded:

» The performance of LLVM/Clang 3.3 for most tests is at least comparable to GCC «

Nobu from their Forums supplied a conclusion which represents the data much better:

» GCC is much faster in anything which uses OpenMP, and moderately faster or equal in anything (except compile times) which doesn't [use OpenMP] «

But Michael from Phoronix did not stop at just ignoring the performance difference between GCC and LLVM. He went on claiming, that

In a few benchmarks LLVM/Clang is faster, particularly when it comes to build times.

And this is blatant reality-distortion which I am very tempted to ascribe to favoritism. LLVM is not “particularly” faster when it comes to build times.

LLVM on AMD FX-8350 Vishera is faster ONLY when it comes to build times!

This was not the first time that I read data-distorting conclusions on Phoronix - and my complaints about that in their forum did not change their actions. So I hope that my post here can help making them aware that deliberately distorting test results is unacceptable.

For my work, compiler performance is actually quite important, because I use programs which run for days or weeks, so 10% runtime reduction can mean saving several days - not counting the cost of using up cluster time.

To fix their blunders, what they would have to do is:

• Avoiding Benchmarks which only one compiler supports properly (OpenMP).
• Marking the compile time tests explicitely, so they strongly stand out from the rest, because they measure a completely different parameter than the other tests: Compiler Runtime vs. Performance of the Compiled Binaries.
• Writing conclusions which actually fit their results.

Their current approach gives a distinct disadvantage to GCC (even for the OpenMP tests, because they convey the notion that if LLVM only had OpenMP, it would be better in everything - which as this test shows is simply false), so the compiler-tests from Phoronix work as covert propaganda against GCC, even in tests where GCC flat-out wins. And I already don’t like open propaganda, but when the propaganda gets masked as objective testing, I actually get angry.

I hope my post here can help move them towards doing proper testing again.

PS: I write so strongly here, because I actually like the tests from Phoronix a lot. I think we need rather more than less testing and their testsuite actually seems to do a good job - when given the right parameters - so seeing Phoronix distorting the tests to a point where they become almost useless (except as political tool against GCC) is a huge disappointment to me.

pyRad - a wheel type command interface for KDE

pyRad is a wheel type command interface for KDE¹, designed to appear below your mouse pointer at a gesture.

install | setup | usage and screenshots | download and sources

Install

in any distro

• Get Python.
• call easy_install pyRadKDE in any shell.
• Test it by calling pyrad.py.

• This should automatically pull in pyKDE4. If it doesn’t, you need to install that seperately.

• For a "live" version, just clone the pyrad Mercurial repo and let KDE run "path/to/repo/pyrad.py" at startup. You can stop a running pyrad via pyrad.py --quit. pyrad.py --help gives usage instructions.

In Gentoo

• emerge -a kde-misc/pyrad

In unfree systems (like MacOSX and Windows)

• I have no clue since I don’t use them. You’ll need to find out yourself or install a free system. Examples are Kubuntu for beginners and Gentoo for convenient tinkering. Both run GNU/Linux.

Setup

• Run /usr/bin/pyrad.py. Then add it as script to your autostart (systemsettings→advanced→autostart). You can now use Alt-F6 and Meta-F6 to call it.

Mouse gesture (optional)

• Add the mouse gesture in systemsettings (systemsettings→shortcuts) to call D-Bus: Program: org.kde.pyRad ; Object: /MainApplication ; Function: newInstance (you might have to enable gestures in the settings, too - in the shortcuts-window you should find a settings button).

• Alternately set the gesture to call the command dbus-send --type=method_call --dest=org.kde.pyRad /MainApplication org.kde.KUniqueApplication.newInstance.

Customize the wheel

Customize the menu by editing the file "$HOME/.pyradrc" or middle-clicking (add) and right-clicking (edit) items.

Usage and screenshots

To call pyRad and see the command wheel, you simply use the gesture or key you assigned.

Then you can activate an action with a single left click. Actions can be grouped into folders. To open a folder, you also simply left-click it.

Also you can click the keyboard key shown at the beginning of the tooltip to activate an action (hover the mouse over an icon to see the tooltip).

To make the wheel disappear or leave a folder, click the center or hit the key 0. To just make it disappear, hit escape.

For editing an action, just right click it, and you’ll see the edit dialog.

Each item has an icon (either an icon name from KDE or the path to an icon) and an action. The action is simply the command you would call in the shell (only simple commands, though, no real shell scripting or glob).

To add a new action, simply middle-click the action before it. The wheel goes clockwise, with the first item being at the bottom. To add a new first item, middle-click the center.

To add a new folder (or turn an item into a folder), simply click on the folder button, say OK and then click it to add actions in there.

See it in action:

download and sources

pyRad is available from

• kde-apps,
• PyPI (file releases) and
• bitbucket (source repository for contributing)

PS: The name is a play on ‘python’, ‘Rad’ (german for wheel) and pirate :-)

PPS: KDE, K Desktop Environment and the KDE Logo are trademarks of KDE e.V.

PPPS: License is GPL+ as with almost everything on this site.

Python for beginning programmers

(written on ohloh for Python)

Since we already have two good reviews from experienced programmers, I'll focus on the area I know about: Python as first language.

My experience:

• I began to get into coding only a short time ago. I already knew about processes in programs, but not how to get them into code.
• I wanted to learn C/C++ and failed at general structure. After a while I could do it, but it didn't feel right.
• I tried my luck with Java and didn't quite get going.
• Then I tried Python, and got in at once.

Advantages of Python:

• The structure of programs can be understood easily.
• The Python interpreter lets you experiment very quickly.
• You can realize complex programs, but Python also allows for quick and simple scripting.
• Code written by others is extremely readable.
• And coding just flows - almost like natural speaking/thinking.

As a bonus, there is the great open book How to Think Like a Computer Scientist which teaches Python and is being used for teaching Python and Programming at universities.

So I can wholeheartedly recommend Python to beginners in programming, and as the other reviews on Ohloh show, it is also a great language for experienced programmers and seems to be a good language to accompany you in your whole coding life.

PS: Yes, I know about the double meaning of "first language" :)

Reducing the Python startup time

The python startup time always nagged me (17-30ms) and I just searched again for a way to reduce it, when I found this:

The Python-Launcher caches GTK imports and forks new processes to reduce the startup time of python GUI programs.

Python-launcher does not solve my problem directly, but it points into an interesting direction: If you create a small daemon which you can contact via the shell to fork a new instance, you might be able to get rid of your startup time.

To get an example of the possibilities, download the python-launcher and socat and do the following:

PYTHONPATH="../lib.linux-x86_64-2.7/" python python-launcher-daemon &
echo pass > 1
for i in {1..100}; do 
    echo 1 | socat STDIN UNIX-CONNECT:/tmp/python-launcher-daemon.socket & 
done

Todo: Adapt it to a given program and remove the GTK stuff. Note the & at the end: Closing the socket connection seems to be slow, so I just don’t wait for socat to finish. Breaks at somewhere over 200 simultaneous connections. Option: Use a datagram socket instead.

The essential trick is to just create a server which opens a socket. Then it reads all the data from the socket. Once it has the data, it forks like the following:

        pid = os.fork()
        if pid:
            return

        signal.signal(signal.SIGPIPE, signal.SIG_DFL)
        signal.signal(signal.SIGCHLD, signal.SIG_DFL)

        glob = dict(__name__="__main__")
        print 'launching', program
        execfile(program, glob, glob)

        raise SystemExit

Running a program that way 100-times took just 0.23 seconds for me so the Python startup time of 17ms got reduced to 2.3ms.

You might have to switch from forking to just executing the code instead of forking if you want to be even faster and the code snippets are small. For example when running the same test without the fork and the signals, 100 executions of the same code took just 0.09s, cutting down the startup time to an impressing 0.9ms - with the cost of no longer running in parallel.

(That’s what I also do with emacsclient… My emacs takes ~30s to start (due to excessive use of additional libraries I added), but emacsclient -c shows up almost instantly.)

I tested the speed by just sending a file with the following snippet to the server:

import time
with open("2", "a") as f:
    f.write(str(time.time()) + "\n")

Note: If your script only needs the included python libraries (batteries) and no custom-installed libs, you can also reduce the startuptime by avoiding site initialization:

python -S [script]

Without -S python -c '' takes 0.018s for me. With -S I am down to

time python -S -c '' → 0.004s. 

Note that you might miss some installed packages that way. This is slower than the daemon method by up to factor 4 (4ms instead of 0.9), but still faster than the default way. Note that cold disk buffers can make the difference much bigger on the first run which is not relevant in this case but very much relevant in general for the impression of startup speed.

PS: I attached the python-launcher 0.1.0 in case its website goes down. License: GPL and MIT; included. This message was originally written at stackoverflow.

Screencast: Tabbing of everything in KDE

I just discovered tabbing of everything in KDE:

(download)

Created with recordmydesktop, cut with kdenlive, encoded to ogg theora with ffmpeg2theora (encoding command).

Music: Beat into Submission on Public Domain by Tryad.

To embed the video on your own site you can simply use:

<video 
src="http://draketo.de/files/screencast-tabbing-everywhere-kde.ogv"
controls=controls>
</video>

If you do so, please provide a backlink here.

License: cc by-sa, because that’s the license of the song. If you omit the audio, you can also use one of my usual free licenses (or all of them, including the GPL). Here’s the raw recording (=video source).

¹: Feel free to upload the video to youtube or similar. I license my stuff under free licenses to make it easy for everyone to use, change and spread them.

²: Others have shown this before, but I don’t mind that. I just love the feature, so I want to show it :)

³: The command wheel I use for calling programs is the pyRad.

Shackle-Feats: The poisoned Apple

This is a mail I sent as listener comment to Free as in Freedom.

Hi Bradley, Hi Karen,

I am currently listening to your Steve Jobs show (yes, late, but time is scarce these days).

And I side with Karen (though I use KDE): Steve Jobs managed to make a user interface which feels very natural. And that is no problem in itself. Apple solved a problem: User interfaces are hard to use for people who don’t have computer experience and who don’t have time to learn using computers right.

But they then used that solution to lure people into traps they set up to get our money and our freedom.

As analogy: A friend of mine told me, that Photoshop gives her Freedom, because she can do things with it, which she can’t do with anything else. And she’s right on that: She gets a kind of freedom. But she has to give up other freedoms for that, for example the freedom to do freelancing work without paying 3000€ up front.

To make the problem with that kind of freedom visible, let’s use one more analogy: When I get a flying car with which I can visit the Himalaya without having to get a drivers license, then I just got the Freedom to actually visit Himalaya. But sadly that car comes with a rule, that I am not allowed to take friends with me, and it does not allow me to drive into cities ruled by left-wing politicians. It costs so much, that I can’t afford another car¹, so now if I want to visit Himalaya, I can’t take friends with me even when I just want to drive to the next shop and I can’t visit left-wing friends.

That car would give me a kind of Freedom, but it would take away other freedoms I had before I used it. If all people used it, the effects would be horrible, and not just for left-wings and car owners: You would not be able to get a ride from a neighbor when you needed to get to the doctor fast.

Now imagine what would happen, if people would find ways to make money with that flying car. They would create a society, where you have to give up Freedom if you want to get one of the good jobs.

So creating a new kind of Freedom and coupling it with heavy shackles does not give you more Freedom. It creates a situation where people have a harder time living their life when they want to keep their basic freedom, because those shackle-feats become mandatory.

I need to remember the name shackle-feats :)

Apple kinda invented the shackle-feat “use shiny computers without understanding them”.
They managed to make shackles almost mandatory for parts of society by creating a pressure on people that they have to be able to do the feat, so they have to accept the shackles.

Now we have to recreate that feat without the shackles so people are able to keep up without losing their freedom. We have to do additional work, because society is being shaped by those who made the shackles.

Best wishes,
Arne Babenhauserheide

PS: Steve Jobs managed to create really nice interfaces. Sadly he used his abilities to shackle people. He once was a hero to me. Even today there is stuff he did that I admire. But he decided to use his abilities for shackling people.

Simple positive trust scheme with threshholds

I don’t see a reason for negative reputation schemes — voting down is in my view a flawed concept.

The rest of this article is written for freetalk inside freenet, and also posted there with my nonanonymous ID.

That just allows for community censorship, which I see as incompatible with the goals of freenet.

Would it be possible to change that to use only positive votes and a threshhold?

• If I like what some people write, I give them positive votes.
• If I get too much spam, I increase the threshhold for all people.
• Effective positive votes get added. It suffices that some people I trust also trust someone else and I’ll see the messages.
• Effective trust is my trust (0..1) · the trust of the next in the chain (0..1) · …

Usecase:

• Zwister trusts Alice and Bob.
• Alice trusts Lilith.
• Bob hates Lilith.

In the current scheme (as I understand it), zwister wouldn’t see posts from Lilith.

In a pure positive scheme, zwister would see the posts. If zwister wants to avoid seeing the posts from Lilith, he has to untrust Alice or ask Alice to untrust Lilith. Add to that a personal (and not propagating) blocking option which allows me to “never see anything from Lilith again”.

Bob should not be able to interfere with me seeing the messages from Lilith, when Alice trusts Lilith.

If zwisters trust for Alice (0..1) multiplied with Alices trust for Lilith (0..1) is lower than zwisters threshhold, zwister doesn’t see the messages.

PS: somehow adapted from Credence, which would have brought community spam control to Gnutella, if Limewire had adopted it.

PPS: And adaption for news voting: You give positive votes on news which show up. Negative votes assign a private threshhold to the author of the news, so you then only see news from that author which enough people vote for.

Simple steps to attach the GNU Public License (GPL) to your project

Here's the simple steps to attach a GPL license to your source files (written after requests by DiggClone and Bandnet):

For your own project, just add the following text-notice to the header/first section of each of your source-files, commented out in whatever way your language uses:

----------------following is the notice-----------------
/*
* Your Project Name - -you slogan-
* Copyright (C) 2007 - 2007 Your Name
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
*/
----------------------------------------------
the "2007 - 2007" needs to be adjusted to "year when you gave it the license in the first place" - "current year".

Then put the file gpl.txt into the source-folder or a docs folder: http://www.gnu.org/licenses/gpl.txt

If you are developing together with other people, you need their permission to put the project under the GPL.

------

Just for additional Info, I found this license comparision paper by sun: http://mediacast.sun.com/share/webmink/SunLicensingWhitePaper042006.pdf

And comments to it: http://blogs.sun.com/webmink/entry/open_source_licensing_paper#comments

It does look nice, but it misses one point:

GPL is trust: Contributors can trust, that their contributions will keep helping the community, and that the software they contribute to will keep being accessible for the community.

(That's why I decided some years ago to only support GPL projects. My contributions to one semi-closed project got lost, because the project wasn't free and the developer just decided not to offer them anymore, and I could only watch hundreds of hours of work disappear, and that hurt.)

Best wishes,
Arne
PS: If anything's missing, please write a comment!

Some Python Programs of mine

heavily outdated page. See bitbucket.org/ArneBab for many more projects…

Hi,

I created some projects with pyglet and some tools to facilitate 2D
game development (for me), and I though you might be interested.

• babglet: basic usage of pyglet for 2D games with optional collision
  detection and avoidance.

• blob_swarm: a swarm of blobs with emerging swarm behaviour through only pair relations.
• blob_battle: a duel-style battle between two blobs (basic graphics,
  control and movement done)

• fuzzy_collisions: 2 groups of blobs. One can be controlled. When two
  blobs collide, they move away a (random) bit to avoid the collision.

They are avaible from the rpg-1d6 project on sourceforge:
-> https://sf.net/projects/rpg-1d6/

The download can be found at the sf.net download page:
-> https://sourceforge.net/project/showfiles.php?group_id=199744

Steve Jobs, Get Your Head out Of the Sand! - Broken Apple Heart

Dear Steve,

The problem for DRM proponents is that DRM doesn't work as advertised - and you are helping perpetuate a lie. We know you know this, you've said as much about music and DRM yourself. So why do you persist in touting DRM for video?

What DRM does do is trample my rights and create a situation where, if I were to circumvent a DRM scheme to be in control of my computer, it would be a criminal act - thanks to legislation like the Digital Millennium Copyright Act (DMCA).

Do you still have Apple's head stuck in the sand?
I'm writing to suggest you take it out.
- http://defectivebydesign.com

Personal Comment:
I've been a Mac User my whole life. I left you with a broken heart, when you used the TPM chip to lock down _developer_ Macs.

Now I'm a GNU/Linux user (KDE), and even though I sometimes think back to Macs, to Shufflepuck (my first addiction), to professional video editing (With my old 66Mhz Mac), to the 6 months when I tried every beta of MacOSX even though my 266Mhz G3 was far too slow to render it in speed, and to the ease of music production on my Flat-Panel iMac, I won't come back to have my freedom taken.

You're creating great computers. Why do you still have to make them a tool for digital slavery, even though you now aknowledged yourself, that this slavery is bad?

The one who acts bad but doesn't know it, is a fool.
The one who acts bad and knows it, is a criminal, regardless of the laws.

Disappointed wishes with but a glimpse of hope,
Arne Babenhauserheide - Broken Apple Heart (german)

Tail Call Optimization (TCO), dependency, broken debug builds in C and C++ — and gcc 4.8

TCO: Reducing the algorithmic complexity of recursion.
Debug build: add overhead to a program to trace errors.

UPDATE: GCC 4.8 gives us -Og -foptimize-sibling-calls, and I had a few quite embarrassing errors in my C - thanks to AKF for the catch!

1 Intro

Tail Call Optimization (TCO) makes this

def foo(n):
    print(n)
    return foo(n+1)

foo(1)

behave like this

def foo(n):
    print(n)
    return n+1

n = 1
while True:
    n = foo(n)

2 The tests

(define (foo n)
  (display n)
  (newline)
  (foo (1+ n)))

(foo 1)

def foo(n):
    print n
    return foo(n+1)

foo(1)

#include <stdio.h>

int recurse(int n)
{
  printf("%i\n", n);
  return recurse(n+1);
}

int main()
{
  return recurse(1);
}

all : C2 Cg Cg2 C1

# optimized
C2 : tailcallc.c
    g++ -O2 tailcallc.c -o C2

# debug build
Cg : tailcallc.c
    g++ -g tailcallc.c -o Cg

# optimized debug build
Cg2 : tailcallc.c
    g++ -g -O2 tailcallc.c -o Cg2

# only slightly optimized
C1 : tailcallc.c
    g++ -O1 tailcallc.c -o C1

Traceback (most recent call last):
 File "tailcallpython.py", line 6, in <module>
   foo(1)
 File "tailcallpython.py", line 4, in foo
   return foo(n+1)
… repeat about 997 times …
RuntimeError: maximum recursion depth exceeded

g++ -O2 C.c -o C2
./C2

g++ -g C.c -o Cg
./Cg 

7228 KB   [stack]
56 KB [heap]
40 KB /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6.0.17
24 KB /lib64/libc-2.15.so
12 KB /home/arne/.emacs.d/private/journal/Cg

352 KB    /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libstdc++.so.6.0.17
252 KB    /lib64/libc-2.15.so
108 KB    /lib64/ld-2.15.so
60 KB /lib64/libm-2.15.so
16 KB /usr/lib64/gcc/x86_64-pc-linux-gnu/4.7.2/libgcc_s.so.1

recurse (n=43) at C.c:5
5         printf("%i\n", n);
43
6         return recurse(n+1);

g++ -g -O2 C.c -o Cg2
./Cg2

5         printf("%i\n", n);
printf (__fmt=0x40069c "%i\n") at /usr/include/bits/stdio2.h:105
105       return __printf_chk (__USE_FORTIFY_LEVEL - 1, __fmt, __va_arg_pack ());
5
6         return recurse(n+1);

g++ -g -O1 -foptimize-sibling-calls C.c -o Cgtco
./Cg 

g++ -g -Og -foptimize-sibling-calls C.c -o Cgtco
./Cgtco 

recurse (n=n@entry=1) at C.c:4
4       {
5         printf("%i\n", n);
1
6         return recurse(n+1);
5         printf("%i\n", n);
2
6         return recurse(n+1);

g++ -O1 C.c -o C1
./C1

The danger of promoting dead closed clients

I had a strange feeling about people advertising the dead and closed source Gnutella client BearShare, but I only found one of the reasons for that gut feeling today.

Assumptions I use: We want Gnutella to continue to evolve and grow better.

To have Gnutella evolve, the developers of actively developed clients need feedback (and be it only encouragement).

If people now use a dead client, which won't evolve anymore, they don't provide essential feedback to actively developed clients, and it might even happen, that some developers waste time on trying to hack the dead client to make something work (again), instead of contributing to an active open client.

So every user who uses a dead closed client instead of an active open (and free) client hinders the evolution of Gnutella.

That's not the fault of the user, and it's not per se damaging to the current state of the network (as long as the user shares, he contributes to the availaable files), but on the long term it hinders Gnutella from becoming better.

And that in mind, promoting a closed dead client directly damages Gnutella.

I know I'm human and as such prone to errors, so if you see anything I overlooked, please tell me about it.

The dynamics of free culture and the danger of noncommercial clauses

NC covered works trick people into investing in a dead end

Free licensing lowers the barrier of entry to creating cultural works, which unlocks a dynamic where people can realize their ideas much easier - and where culture can actually live, creating memes, adjusting them to new situations and using new approaches with old topics.

But for that to really take off, people have to be able to make a living from their creations - which build on other works. Then we have people who make a living by reshaping culture again and again - instead of the current culture where only a few (rich or funded by rich ones) can afford to reuse old works and all others have to start from scratch again and again.

Sharealike licensing gives those who allow others to reuse their works an edge over those who do not do that: They can access many resources early in their career which allow them to produce high-quality stuff without needing to pay huge amounts up front. And they hone their skills in working with free stuff. So when they become good enough that they can work in art for a living, they are deeply invested in free culture, so they have very good reasons for also licensing their new works under free licenses.

As a real-life example for the dynamic of free licensing, I’ve been working on a free tabletop roleplaying system in my free time for the last 10 years. For 3 or 4 years now it has been licensed under the GPL, so we could use images from Battle for Wesnoth in our books. And 2 years ago, I worked together with another roleplayer to create minimal roleplaying supplements on just one Flyer - where only half the images were from Battle for Wesnoth, because a great artist decided to contribute (All hail Trudy!).

All this would have been possible with NC licensing.

But about 2 months ago a roleplayer from a forum I discuss at unveiled his plans to create a german free rpg day and I realized that our minimal RPG would be a great fit for that - but that I could not afford myself to print it in high enough numbers and good enough quality to reach many people.

So I worked on the design and text to polish them, and when I was happy I started a 4-day fundraiser to finance printing the RPGs. Within just those 4 days I got over 200€ in donations which allowed me to print 2000 RPGs in great quality along with supplements and additional character cards which made every single RPG instantly playable - instead of 1500 RPGs with only one card so people would need 3 RPGs to actually play.

And this would have been plain illegal with NC material.

It is not yet “making a living with free art”, but it is a first step out of the purely hobby creation into a stronger dynamic. One which allows us to bring 2000 physical RPGs to people without going broke - and more importantly: One which started small and can grow organically.

An RPG might not be the best example here, because tabletop RPGs are notoriously bad for generating money. But it is the example I experienced myself.

As an example which might be closer to you: Imagine that you created a movie with free music and other material from free licensed works. Imagine that half of the visuals you use could have already been created - maybe for some other movie. By using free stuff, you could save half the effort for creating the movie.

But if that other stuff had been NC, you would not be allowed to start a fundraiser for getting it to blu-ray quality - at least not without replacing all NC parts, which would have added a high cost to be able to increase your outreach. Likely it would have been a blocking cost. It would have been easier to just create a new project than to polish the one you have to reach more people.

And polish is what allowed me to move the RPG from just being a hardly readable PDF to a work I can look at with pride.

To wrap it up: Free culture - just like free software - allows people to take little steps into creating culture and to move organically from just being a hobby artist towards making a living from their work - and spreading their work to many more people.

And NC covered works trick people into investing in a dead end, because they can never move beyond being a hobbyist without huge investments which bring no other benefit than recreating what they could directly use when they did not try to make a living. It’s like learning to use Photoshop and then realizing that you aren’t allowed to earn a little extra by improving wedding-images without shelling out 3000€ for a creative suite license. And that means, that you can’t move in small steps from a boring day job to a professional creative life.

(written in reply to a question from Keith, one of the makers of Software Wars, a movie about free software which is trying to fund going to a high-quality blu-ray release at the moment)

The effect of the optional restrictions of the GPLv3

I just thought a bit about the restrictions the GPLv3 allows, and I think I just understood their purpose and effect for the first time (correct me, if I'm wrong :) ).

What are the restrictions?

The GPLv3 allows developers (=copyright holders) to add selected restrictions, like forbidding the use of a certain brand name or similar.

The catch with them is, that any subsequent developer who adds anything is free to simply strip off the restrictions.

What is their effect?

Now I wondered for a long time, what that really gains us. today I then realized that subsequent develoeprs are only free to strip off the restrictions, as long as that doesn't violate any license of some part of the program.

That means, the GPLv3 restrictions simply have the effect of adding compatibility to other licenses, while keeping the option to strip off any restriction, when you replace the part under the other license with a more liberal licensed part.

So this doesn't place any additional burden on packagers, because they already have to check those other licenses for their restrictions. Now the GPLv3 description of the whole package clearly states what additional restrictions are inferred by the parts which are under different but compatible licenses.

While those parts where under seperate licenses before (and had to be checked), they can now be impcoved with GPLv3 code with additional restrictions.

And as soon as the GPLv3 code can stand on its own feet, the more restrictive licensed part can be replaces with GPLv3 code, and the restrictions can be removed again, making the work of the packagers easier.

Better still, the GPLv3 shows clearly the sum of all restrictions of the individual (differently but compatible licensed) parts, so packagers only need to check the GPLv3 license information to see all restrictions in a standardized format (GPLv3 additional restrictions).

Example

Let's assume I find this great piece of software which says "do what you want but don't touch my brand", and I want to build my GPLv3 program on it. Let's call the piece of software "foo". So I just begin coding and use the GPLv3 for my parts (simply copyright message in my code files). For the whole package I add a license information ("license.txt" or "COPYING" or similar) which give the information

  "This program is licensed under the GPLv3 
  with the additional restriction that the brand 
  'foo' may not be used for derived products. 
  The additional restriction is inferred by the package foo.

  (plus license mumbo jumbo you can find at and copy 
  from http://gnu.org/licenses/gpl.html)"

Now someone else takes my program and improves it. But he also uses the package "blah" which also says that it's brand must not be violated. Now the combined license would be:

  "This program is licensed under the GPLv3 
  with the additional restriction that the brand 
  'foo' and the brand 'blah' may not be used for 
  derived products. 
  The additional restriction for brand 'foo' is inferred by the package foo.
  The additional restriction for brand 'blah' is inferred by the package blah. 

  (plus license mumbo jumbo you can find at and copy 
  from http://gnu.org/licenses/gpl.html)"

So now a group of free software activists takes offense at the restrictions. They don't want anyone to be restricted by copyright from using a brand. One reason could be that the brand protection was voided by some trademark action.

Now they can't just say "that brand isn't protected anymore", since the protection was reinforced by copyright law.

But they can just replace the parts under the more restrictive licenses with honest GPLv3 licensed parts - either by writing them or by finding a drop-in replacement. Now let's suppose they write the packages "bar" and "baz" which implement the functionality of "foo" and "blah". They now no longer use any parts under licenses which require additional restrictions, so they are free to remove them. As they release their package, the license information might read as follows:

  "This program is licensed under the GPLv3.  

  (plus license mumbo jumbo you can find at and copy 
  from http://gnu.org/licenses/gpl.html)"

If they are nice (and we assume they are) their changelog will also contain a line saying something like "replaced 'foo' and 'blah' which allowed us to remove the additional license restrictions to avoid using the brands 'foo' or 'blah'."

Final remark

As you see (and if I understand it correctly), the additional restrictions can be a great tool for freeing software from restrictions, because they allow you to combine GPlv3 code with somewhat restrictive licensed code and get rid off the restrictions later by replacing the more restrictive licensed parts.

And since the only allowed additional restrictions are those which don't harm the four freedoms of free software, you can still make sure that you use ethically sound software by simply checking whether it is GPL licensed.

So kudos to the designers of the GPLv3. They did such a great job that it took me two years to realize one of the many powerful tools they gave us with the GPLv3 - and I did take part in the public discussion of the GPLv3 since draft 1 (but I never watched a GPLv3 speech...).

Also, since the GPLv3 allows combination with the AGPLv3 software (which adds the restriction, that the source code must also be supplied when the software is used over a network), it gives us a clear path into the future where people might use more and more software "as a service", so it doesn't get executed on their local machine and the normal GPL alone isn't enough to protect our freedom.

The generation of cultural freedom

I am part of a generation who experienced true cultural freedom - and who experienced that freedom being destroyed.

We had access to the largest public library which ever existed and saw it burned down for lust for control.

Not even for greed or gain, because enough studies showed that we did no damage and that we actually paid more for cultural goods than those who did not enjoy that freedom. They fought for control over us.

And the loss of cultural freedom is only the precursor for the loss of personal freedom, as those many new censorship laws show.

I decided to become active to stop that, but there was always one thing I wondered about:

Who are the persons who want internet censorship, PIPA, ACTA, etc?

Not the companies, nor the governments. The top proponents in the background. Those who make sure that basically the same laws are proposed over and over again by different strawmen and in different places.

(my question to the RightsCon Rio - send yours)

The “Apple helps free software” myth

→ Comment to “apple supports a number of opensource projects. Webkit and CUPS come to mind”.

Apple supports a number of copyleft projects, because they have to. They chose to profit from the work other people released as copyleft, and so they are obliged to release their improvements.

Webkit

Webkit is an especially good example of this: Apple took the khtml code from KDE, worked with it for half a year and only released binaries (which is a breach of the license of khtml) until they finally released their code in one big code-drop which the khtml folks had no chance of integrating cleanly.

That way Apple broke away from the community and created their own fork in a way which made sure that the KDE folks could not profit from Apples work without throwing out their own structure.

They still had to adhere the license, though, which enabled others to use Webkit - and essentially created a revolution in Webbrowser-development, because Apple added all the polish needed for a modern browser. If you look at the way they treated the khtml developers, though, do you really think they would have released any code on that critical part of their OS, if they had not been forced to do so by the strong copyleft used by KDE?

Cups

CUPS, the other example of Apple-maintained free software, … is GPL licensed, too. No surprise there: Why else should Apple give their work to others, if not because the license forces them to?

And even there they try to get out by adding a GPL-exception to the parts they write, which allows using those parts without giving out source code. But “This exception is only available for Apple OS-Developed Software and does not apply to software that is distributed for use on other operating systems”.

What do you think how much they will still maintain, as soon as they managed to get that header into all files - and don’t fear a free fork anymore? (also note, that shortly after Apple started maintaining cups, it broke on my GNU/Linux system - „Ein Schelm, wer böses dabei denkt“, as we say in Germany)

Darwin

Just look at what they did with Darwin. They took all the code from FreeBSD. Then they kept the uninteresting part free as long as needed to have a good name and get people work in their spare time on porting it to intel architectures, a work which greatly benefitted Apple, because they could then get away from PPC to no longer depend on IBM. The interesting part however, the graphical interface, was completely locked up from the beginning.

See why OpenDarwin stopped: “Availability of sources, interaction with Apple representatives, difficulty building and tracking sources, and a lack of interest from the community“ — OpenDarwin Shutting Down

4 of 5 reasons for stopping the free alternative directly come from Apple…

Epilogue

Should I complain about that? Actually no. After all, they are allowed to do it by the license. They just do what they can to maximize their monetary gain.

And actually I prefer seeing a big company use copyleft programs to improve its products, because that means that others will be able to achieve at least that part with free software.

If I should complain about anybody, then about all the people who praise Apple for doing what they are forced to do to get the work of others for free - and about shortsighted developers, who use non-copyleft licenses, which allow folks like Apple to save lots of money while locking out others and creating “the computer as a jail made cool”, as Richard M. Stallman put it quite nicely — I call that shackle-feats.

turn files with wikipedia syntax to html (simple python script using mediawiki api)

I needed to convert a huge batch of mediawiki-files to html (had a 2010-03 copy of the now dead limewire wiki lying around). With a tip from RoanKattouw in #mediawiki@freenode.net I created a simple python script to convert arbitrary files from mediawiki syntax to html.

Usage:

• Download the script and install the dependencies (yaml and python 3).
• ./parse_wikipedia_files_to_html.py <files>

This script is neither written for speed or anything (do you know how slow a webrequest is, compared to even horribly inefficient code? …): The only optimization is for programming convenience — the advantage of that is that it’s just 47 lines of code :)

It also isn’t perfect: it breaks at some pages (and informs you about that).

It requires yaml and Python 3.x.

#!/usr/bin/env python3

"""Simply turn all input files to html. 
No errorchecking, so keep backups. 
It uses the mediawiki webapi, 
so you need to be online.

Copyright: 2010 © Arne Babenhauserheide
License: You can use this under the GPLv3 or later, 
         if you add the appropriate license files
         → http://gnu.org/licenses/gpl.html
"""

from urllib.request import urlopen
from urllib.parse import quote
from urllib.error import HTTPError, URLError
from time import sleep
from random import random
from yaml import load
from sys import argv

mediawiki_files = argv[1:]

def wikitext_to_html(text):
    """parse text in mediawiki markup to html."""
    url = "http://en.wikipedia.org/w/api.php?action=parse&format=yaml&text=" + quote(text, safe="") + " "
    f = urlopen(url)
    y = f.read()
    f.close()
    text = load(y)["parse"]["text"]["*"]
    return text

for mf in mediawiki_files:
    with open(mf) as f:
        text = f.read()
    HTML_HEADER = "<html><head><title>" + mf + "</title></head><body>"
    HTML_FOOTER = "</body></html>"
    try: 
        text = wikitext_to_html(text)
        with open(mf, "w") as f:
            f.write(HTML_HEADER)
            f.write(text)
            f.write(HTML_FOOTER)
    except HTTPError:
        print("Error converting file", mf)
    except URLError:
        print("Server doesn’t like us :(", mf)
        sleep(10*random())
    # add a random wait, so the api server doesn’t kick us
    sleep(3*random())

Weltenwald-theme under AGPL (Drupal)

After the last round of polishing, I decided to publish my theme under AGPLv3. Reason: If you use AGPL code and people access it over a network, you have to offer them the code. Which I hereby do ;)
That’s the only way to make sure that website code stays free.

It’s still for Drupal 5, because I didn’t get around to port it, and it has some ugly hacks, but it should be fully functional.

Just untar it in any Drupal 5 install.

tar xjf weltenwald-theme-2010-08-05_r1.tar.bz2

Maybe I’ll get around to properly package it in the future…

Until then, feel free to do so yourself :)

And should I change the theme without posting a new layout here, just drop me a line and I’ll upload a new version — as required by AGPL. And should you have some problem, or if something should be missing, please drop me a line, too.

No screenshot, because a live version kicks a screenshot any day ;)
(in case it isn’t clear: Weltenwald is the theme I use on this site)

Why free speech does not equal to the right of being heard

→ written in a discussion with Sascha¹ in Freenet using Sone.

If free speech included being allowed to force all people to listen, then it would also include my right to force you to listen to everything I say.

Think this on the scale of 6 billion people all using freenet. Every one of them could force you to listen to him/her/it. Whom would you ignore?

In WoT getting some people² to see your message is possible, but it has a price: solving captchas.

The same is true for real life demonstrations: If you want to be seen, you have to get up and actually invest something - be it time, effort or risk to your reputation.

In real life we have channels through which we sell our attention. They are called advertisements and advertisement financed services, and access to our attention is tightly controlled by some few gatekeepers who make lots of money by keeping a hold on our attention.

In freenet all you have to do is solve some captchas, a rule which is the same for everyone.

Why Gnutella scales quite well

You might have read in some (almost ancient) papers, that a network like Gnutella can't scale. So I want to show you, why the current Version of Gnutella does scale, and does it well.

In earlier versions, up to v0.4, Gnutella was a a pure broadcast network. That means, that every search request did reach every participant, so the number of search requests hitting each node was for an optimal network exactly equal to the number of requests, made by nodes who were in the network. And you can see easily why that can't scale.
But that was only true for Gnutella 0.4.

In the current incarnation of Gnutella (Gnutella 0.6), Gnutella is no longer a pure Broadcast network. Instead, only the smallest percentage of the traffic is done via broadcast.

If you want to read about the methods used to realize this, please have a look at the GnuFU guide (english, german).

Here I want to limit it to the statement, that the first two hops of a search request are governed via Dynamic Querying, which stops the request as soon as it has enough sources (this stops a search as soon as it gets about 250 results), and that the last two hops are governed via the Query Routing Protocol, which ensures, that a search request reaches only those hosts, which can actually have the file (which is only about 5% of the nodes).

So in todays reality, Gnutella is a quite structured and very flexible network.

To scale it, Ultrapeers can increase their number of connections from their current 32 upwards, which makes Dynamic Querying (DQ) and the Query Routing Protocol (QRP) even more effective.

In the case of DQ most queries for popular files will still provide enough results after the same number of clients have been contacted, so increasing the number of connections won't change the network traffic at all which is caused by the first two steps.

In the case of QRP, queries wil still only reach the hosts, which can have the file, and if Ultrapeers are connected to more nodes at the same time (by increasing the number of connections), it will provide more results for each connection, so DQ will stop even earlier than with fewer connections per Ultrapeer.

So Gnutella is now far from a broadcast model, and the act of increasing the size of the Gnutella Network can even increase its efficiency for popular files.

For rare files, QRP kicks in with full force, and even though DQ will likely check all other nodes for content, QRP will make sure that only those nodes are reached, which can have the content, which might be only 0.1% of the net or even far less.

Here, increasing the number of nodes per Ultrapeer means that nodes with rare files are in effect closer to you than before, so Gnutella also gets more efficient when you increase the network size, when rare file searches are your major concern.

So you can see, that Gnutella has become a network, which scales extremly well for keyword searches, and due to that it can also very efficiently be used to search for metadata and similar concepts.

The only thing which Gnutella can't do well are searches for strings which aren't seperate words (for example file-hashes), because that kills QRP, so they will likely not reach (m)any hosts. For these types of searches, the Gnutella developers work on a DHT (Distributed Hash Table), which will only be used, if the string can't be split into seperate words, and that DHT will most likely be Kademlia, which is also proven to work quite well.

And with that, the only problem which remains in need of fixing is spam, because that inhibits DQ when you do a rare search, but I am sure that the devs will also find a way to stop spamming, and even with spam, Gnutella is quite effective and consumes very little bandwidth, when you are acting as a leaf, and only moderate bandwidth when you are acting as ultrapeer.

Some figures as finishing touch:

• Leaf network traffic: About 1kB/s if you add outgoing and incoming traffic, which is about the seventh part of the speed of a 56k modem.
• Ultrapeer traffic: About 7kB/s, outgoing and incoming added together, which is about one full ISDN line of less than 1/8th of a DSLs outgoing speed.

Have fun with Gnutella!
- ArneBab 08:14, 15. Nov 2006 (CET)