wisp: Whitespace to Lisp: An indentation to parentheses preprocessor to get more readable Lisp

I love the syntax of Python, but crave the simplicity and power of Lisp.
display "Hello World!"      ↦    (display "Hello World!")
define : hello-world        ↦    (define (hello-world)
  display "Hello World!"    ↦      (display "Hello World!"))
Update (2014-11-19): wisp v0.8.1 released with reader bugfixes. To test it, install Guile 2.0.x and Python 3 and bootstrap wisp:
wget https://bitbucket.org/ArneBab/wisp/downloads/wisp-0.8.1.tar.gz;
tar xf wisp-0.8.1.tar.gz ; cd wisp-0.8.1/;
./configure; make check;
guile -L . --language=wisp tests/factorial.w; echo
If it prints 120120 (two times 120, the factorial of 5), your wisp is fully operational.
That’s it - have fun with wisp syntax!
Update (2014-11-06): wisp v0.8.0 released! The new parser now passes the testsuite and wisp files can be executed directly. For more details, see the NEWS file. To test it, install Guile 2.0.x and bootstrap wisp:
wget https://bitbucket.org/ArneBab/wisp/downloads/wisp-0.8.0.tar.gz;
tar xf wisp-0.8.0.tar.gz ; cd wisp-0.8.0/;
./configure; make check;
guile -L . --language=wisp tests/factorial.w;
echo
If it prints 120120 (two times 120, the factorial of 5), your wisp is fully operational.
That’s it - have fun with wisp syntax!
On a personal note: It’s mindboggling that I could get this far! This is actually a fully bootstrapped indentation sensitive programming language with all the power of Scheme underneath, and it’s a one-person when-my-wife-and-children-sleep sideproject. The extensibility of Guile is awesome!
Update (2014-10-17): wisp v0.6.6 has a new implementation of the parser which now uses the scheme read function. `wisp-scheme.w` parses directly to a scheme syntax-tree instead of a scheme file to be more suitable to an SRFI. For more details, see the NEWS file. To test it, install Guile 2.0.x and bootstrap wisp:
wget https://bitbucket.org/ArneBab/wisp/downloads/wisp-0.6.6.tar.gz;
tar xf wisp-0.6.6.tar.gz; cd wisp-0.6.6;
./configure; make;
guile -L . --language=wisp
That’s it - have fun with wisp syntax at the REPL!
Caveat: It does not support the ' prefix yet (syntax point 4).
Update (2014-01-04): Resolved the name-clash together with Steve Purcell und Kris Jenkins: the javascript wisp-mode was renamed to wispjs-mode and wisp.el is called wisp-mode 0.1.5 again. It provides syntax highlighting for Emacs and minimal indentation support via tab. You can install it with `M-x package-install wisp-mode`
Update (2014-01-03): wisp-mode.el was renamed to wisp 0.1.4 to avoid a name clash with wisp-mode for the javascript-based wisp.
Update (2013-09-13): Wisp now has a REPL! Thanks go to GNU Guile and especially Mark Weaver, who guided me through the process (along with nalaginrut who answered my first clueless questions…).
To test the REPL, get the current code snapshot, unpack it, run ./bootstrap.sh, start guile with $ guile -L . (requires guile 2.x) and enter ,language wisp.
Example usage:
display "Hello World!\n"
then hit enter thrice.
Voilà, you have wisp at the REPL!
Caveeat: the wisp-parser is still experimental and contains known bugs. Use it for testing, but please do not rely on it for important stuff, yet.
Update (2013-09-10): wisp-guile.w can now parse itself! Bootstrapping: The magical feeling of seeing a language (dialect) grow up to live by itself: python3 wisp.py wisp-guile.w > 1 && guile 1 wisp-guile.w > 2 && guile 2 wisp-guile.w > 3 && diff 2 3. Starting today, wisp is implemented in wisp.
Update (2013-08-08): Wisp 0.3.1 released (Changelog).

Table of Contents

2 What is wisp?

Wisp is a simple preprocessor which turns indentation sensitive syntax into Lisp syntax.

The basic goal is to create the simplest possible indentation based syntax which is able to express all possibilities of Lisp.

Basically it works by inferring the brackets of lisp by reading the indentation of lines.

It is related to SRFI-49 and the readable Lisp S-expressions Project (and actually inspired by the latter), but it tries to Keep it Simple and Stupid. Instead of a full alternate reader like readable, it is a simple preprocessor which can be called by any lisp implementation to add support for indentation sensitive syntax.

Just call ./wisp.py –help to see what you can do with it (`./wisp.py -` takes its input from stdin, so it can be used with pipes):

./wisp.py --help
Usage: [-o outfile] [file | -]

Options:
  -h, --help            show this help message and exit
  -o OUTPUT, --output=OUTPUT

Currently wisp is implemented in Python, because that’s the language which I know best and which inspired my wish to use indentation-sensitive syntax in Lisp. To repeat the initial quote:

I love the syntax of Python, but crave the simplicity and power of Lisp.

With wisp I hope to make it possible to create lisp code which is easily readable for non-programmers (and me!) and at the same time keeps the simplicity and power of Lisp.

Its main technical improvements over SRFI-49 and Project Readable are using lines prefixed by a dot (". ") to mark the continuations of the parameters of a function after intermediate function calls and working as a simple preprocessor which can be used with any flavor of Lisp.

The dot-syntax means, instead of marking every function call, it marks every line which does not begin with a function call - which is the much less common case in lisp-code.

3 Wisp syntax rules

  1. A line without indentation is a function call, just as if it would start with a bracket.
    display "Hello World!"      ↦      (display "Hello World!")
    

     
  2. A line which is more indented than the previous line is a sibling to that line: It opens a new bracket.
    display                              ↦    (display
      string-append "Hello " "World!"    ↦      (string-append "Hello " "World!"))
    

     
  3. A line which is not more indented than previous line(s) closes the brackets of all previous lines which have higher or equal indentation. You should only reduce the indentation to indentation levels which were already used by parent lines, else the behaviour is undefined.
    display                              ↦    (display
      string-append "Hello " "World!"    ↦      (string-append "Hello " "World!"))
    display "Hello Again!"               ↦    (display "Hello Again!")
    

     
  4. To add any of ' , or ` to a bracket, just prefix the line with any combination of "' ", ", " or "` " (symbol followed by one space).
    ' "Hello World!"      ↦      '("Hello World!")
    

     
  5. A line whose first non-whitespace characters are a dot followed by a space (". ") does not open a new bracket: it is treated as simple continuation of the first less indented previous line. In the first line this means that this line does not start with a bracket and does not end with a bracket, just as if you had directly written it in lisp without the leading ". ".
    string-append "Hello"        ↦    (string-append "Hello"
      string-append " " "World"  ↦      (string-append " " "World")
      . "!""!")
    

     
  6. A line which contains only whitespace and a colon (":") defines an indentation level at the indentation of the colon. It opens a bracket which gets closed by the next less-indented line. If you need to use a colon by itself. you can escape it as "\:".
    let                       ↦    (let
      :                       ↦      ((msg "Hello World!"))
        msg "Hello World!"    ↦      (display msg))
      display msg             ↦      
    

     
  7. A colon sourrounded by whitespace (" : ") starts a bracket which gets closed at the end of the line.
    define : hello who                    ↦    (define (hello who)
      display                             ↦      (display 
        string-append "Hello " who "!"    ↦        (string-append "Hello " who "!")))
    

     
  8. You can replace any number of consecutive initial spaces by underscores, as long as at least one whitespace is left between the underscores and any following character. You can escape initial underscores by prefixing the first one with \ ("\___ a" → "(___ a)"), if you have to use them as function names.
    define : hello who                    ↦    (define (hello who)
    _ display                             ↦      (display 
    ___ string-append "Hello " who "!"    ↦        (string-append "Hello " who "!")))
    

     

To make that easier to understand, let’s just look at the examples in more detail:

3.1 A simple top-level function call

display "Hello World!"      ↦      (display "Hello World!")

This one is easy: Just add a bracket before and after the content.

3.2 Multiple function calls

display "Hello World!"      ↦      (display "Hello World!")
display "Hello Again!"      ↦      (display "Hello Again!")

Multiple lines with the same indentation are separate function calls (except if one of them starts with ". ", see Continue arguments, shown in a few lines).

3.3 Nested function calls

display                              ↦    (display
  string-append "Hello " "World!"    ↦      (string-append "Hello " "World!"))

If a line is more indented than a previous line, it is a sibling to the previous function: The brackets of the previous function gets closed after the (last) sibling line.

3.4 Continue function arguments

By using a . followed by a space as the first non-whitespace character on a line, you can mark it as continuation of the previous less-indented line. Then it is no function call but continues the list of parameters of the funtcion.

I use a very synthetic example here to avoid introducing additional unrelated concepts.

string-append "Hello"        ↦    (string-append "Hello"
  string-append " " "World"  ↦      (string-append " " "World")
  . "!""!")

As you can see, the final "!" is not treated as a function call but as parameter to the first string-append.

This syntax extends the notion of the dot as identity function. In many lisp implementations1 we already have `(= a (. a))`.

= a        ↦    (= a
  . a      ↦      (. a))

With wisp, we extend that equality to `(= '(a b c) '((. a b c)))`.

. a b c    ↦    a b c

3.5 Double brackets (let-notation)

If you use `let`, you often need double brackets. Since using pure indentation in empty lines would be really error-prone, we need a way to mark a line as indentation level.

To add multiple brackets, we use a colon to mark an intermediate line as additional indentation level.

let                       ↦    (let
  :                       ↦      ((msg "Hello World!"))
    msg "Hello World!"    ↦      (display msg))
  display msg             ↦      

3.6 One-line function calls inline

Since we already use the colon as syntax element, we can make it possible to use it everywhere to open a bracket - even within a line containing other code. Since wide unicode characters would make it hard to find the indentation of that colon, such an inline-function call always ends at the end of the line. Practically that means, the opened bracket of an inline colon always gets closed at the end of the line.

define : hello who                            ↦    (define (hello who)
  display : string-append "Hello " who "!"    ↦      (display (string-append "Hello " who "!")))

This also allows using inline-let, if you only need one variable:

let                       ↦    (let
  : msg "Hello World!"    ↦      ((msg "Hello World!"))
  display msg             ↦      (display msg))

and can be stacked for more compact code:

let : : msg "Hello World!"     ↦    (let ((msg "Hello World!"))
  display msg                  ↦      (display msg))

3.7 Visible indentation

To make the indentation visible in non-whitespace-preserving environments like badly written html, you can replace any number of consecutive initial spaces by underscores, as long as at least one whitespace is left between the underscores and any following character. You can escape initial underscores by prefixing the first one with \ ("\___ a" → "(___ a)"), if you have to use them as function names.

define : hello who                    ↦    (define (hello who)
_ display                             ↦      (display 
___ string-append "Hello " who "!"    ↦        (string-append "Hello " who "!")))

4 Syntax justification

I do not like adding any unnecessary syntax element to lisp. So I want to show explicitely why the syntax elements are required to meet the goal of wisp: indentation-based lisp with a simple preprocessor.

4.1 . (the dot)

We have to be able to continue the arguments of a function after a call to a function, and we must be able to split the arguments over multiple lines. That’s what the leading dot allows. Also the dot at the beginning of the line as marker of the continuation of a variable list is a generalization of using the dot as identity function - which is an implementation detail in many lisps.

`(. a)` is just `a`.

So for the single variable case, this would not even need additional parsing: wisp could just parse ". a" to "(. a)" and produce the correct result in most lisps. But forcing programmers to always use separate lines for each parameter would be very inconvenient, so the definition of the dot at the beginning of the line is extended to mean “take every element in this line as parameter to the parent function”.

Essentially this dot-rule means that we mark variables at the beginning of lines instead of marking function calls, since in Lisp variables at the beginning of a line are much rarer than in other programming languages. In lisp assigning a value to a variable is a function call while it is a syntax element in many other languages, so what would be a variable at the beginning of a line in other languages is a function call in lisp.

(Optimize for the common case, not for the rare case)

4.2 : (the colon)

For double brackets and for some other cases we must have a way to mark indentation levels without any code. I chose the colon, because it is the most common non-alpha-numeric character in normal prose which is not already reserved as syntax by lisp when it is surrounded by whitespace, and because it already gets used for marking keyword arguments to functions in Emacs Lisp, so it does not add completely alien characters.

The function call via inline " : " is a limited generalization of using the colon to mark an indentation level: If we add a syntax-element, we should use it as widely as possible to justify the added syntax overhead.

But if you need to use : as variable or function name, you can still do that by escaping it with a backslash (example: "\:"), so this does not forbid using the character.

4.3 _ (the underscore)

In Python the whitespace hostile html already presents problems with sharing code - for example in email list archives and forums. But in Python the indentation can mostly be inferred by looking at the previous line: If that ends with a colon, the next line must be more indented (there is nothing to clearly mark reduced indentation, though). In wisp we do not have this help, so we need a way to survive in that hostile environment.

The underscore is commonly used to denote a space in URLs, where spaces are inconvenient, but it is rarely used in lisp (where the dash ("-") is mostly used instead), so it seems like a a natural choice.

You can still use underscores anywhere but at the beginning of the line. If you want to use it at the beginning of the line you can simply escape it by prefixing the first underscore with a backslash (example: "\___").

5 Background

A few months ago I found the readable Lisp project which aims at producing indentation based lisp, and I was thrilled. I had already done a small experiment with an indentation to lisp parser, but I was more than willing to throw out my crappy code for the well-integrated parser they had.

Fast forward half a year. It’s February 2013 and I started reading the readable list again after being out of touch for a few months because the birth of my daughter left little time for side-projects. And I was shocked to see that the readable folks had piled lots of additional syntax elements on their beautiful core model, which for me destroyed the simplicity and beauty of lisp. When language programmers add syntax using \\, $ and <>, you can be sure that it is no simple lisp anymore. To me readability does not just mean beautiful code, but rather easy to understand code with simple concepts which are used consistently. I prefer having some ugly corner cases to adding more syntax which makes the whole language more complex.

I told them about that and proposed a simpler structure which achieved almost the same as their complex structure. To my horror they proposed adding my proposal to readable, making it even more bloated (in my opinion). We discussed a long time - the current syntax for inline-colons is a direct result of that discussion in the readable list - then Alan wrote me a nice mail, explaining that readable will keep its direction. He finished with «We hope you continue to work with or on indentation-based syntaxes for Lisp, whether sweet-expressions, your current proposal, or some other future notation you can develop.»

It took me about a month to answer him, but the thought never left my mind (@Alan: See what you did? You anchored the thought of indentation based lisp even deeper in my mind. As if I did not already have too many side-projects… :)).

Then I had finished the first version of a simple whitespace-to-lisp preprocessor.

And today I added support for reading indentation based lisp from standard input which allows actually using it as in-process preprocessor without needing temporary files, so I think it is time for a real release outside my Mercurial repository.

So: Have fun with wisp v0.2 (tarball)!

PS: If you want to run wisp code pseudo-directly, you can use the following script:

#!/bin/sh
~/path/to/wisp.py -o /tmp/wisptmp.scm $@ && guile -l ~/.guile -s /tmp/wisptmp.scm

PPS: Wisp is linked in the comparisions of SRFI-110.

I love the syntax of Python, but crave the simplicity and power of Lisp.display "Hello World!" &#8614; (display "Hello World!")define : hello-world &#8614; (define (hello-world) display "Hello World!" &#8614; (display "Hello World!"))Wisp turns indentation into lisp expressions.Get it from its Mercurial repository:hg clone http://bitbucket.org/ArneBab/wispSee more Examples.Update (2014-11-19): wisp v0.8.1 released with reader bugfixes. To test it, install Guile 2.0.x and Python 3 and bootstrap wisp: wget https://bitbucket.org/ArneBab/wisp/downloads/wisp-0.8.1.tar.gz; tar xf wisp-0.8.1.tar.gz ; cd wisp-0.8.1/; ./configure; make check; guile -L . --language=wisp tests/factorial.w; echo If it prints 120120 (two times 120, the factorial of 5), your wisp is fully operational. That’s it - have fun with wisp syntax!Update (2014-11-06): wisp v0.8.0 released! The new parser now passes the testsuite and wisp files can be executed directly. For more details, see the NEWS file. To test it, install Guile 2.0.x and bootstrap wisp: wget https://bitbucket.org/ArneBab/wisp/downloads/wisp-0.8.0.tar.gz; tar xf wisp-0.8.0.tar.gz ; cd wisp-0.8.0/; ./configure; make check; guile -L . --language=wisp tests/factorial.w; echo If it prints 120120 (two times 120, the factorial of 5), your wisp is fully operational. That’s it - have fun with wisp syntax!On a personal note: It’s mindboggling that I could get this far! This is actually a fully bootstrapped indentation sensitive programming language with all the power of Scheme underneath, and it’s a one-person when-my-wife-and-children-sleep sideproject. The extensibility of Guile is awesome!Update (2014-10-17): wisp v0.6.6 has a new implementation of the parser which now uses the scheme read function. `wisp-scheme.w` parses directly to a scheme syntax-tree instead of a scheme file to be more suitable to an SRFI. For more details, see the NEWS file. To test it, install Guile 2.0.x and bootstrap wisp: wget https://bitbucket.org/ArneBab/wisp/downloads/wisp-0.6.6.tar.gz; tar xf wisp-0.6.6.tar.gz; cd wisp-0.6.6; ./configure; make; guile -L . --language=wisp That’s it - have fun with wisp syntax at the REPL! Caveat: It does not support the ' prefix yet (syntax point 4).Update (2014-01-04): Resolved the name-clash together with Steve Purcell und Kris Jenkins: the javascript wisp-mode was renamed to wispjs-mode and wisp.el is called wisp-mode 0.1.5 again. It provides syntax highlighting for Emacs and minimal indentation support via tab. You can install it with `M-x package-install wisp-mode`Update (2014-01-03): wisp-mode.el was renamed to wisp 0.1.4 to avoid a name clash with wisp-mode for the javascript-based wisp.Update (2013-09-13): Wisp now has a REPL! Thanks go to GNU Guile and especially Mark Weaver, who guided me through the process (along with nalaginrut who answered my first clueless questions…). To test the REPL, get the current code snapshot, unpack it, run ./bootstrap.sh, start guile with $ guile -L . (requires guile 2.x) and enter ,language wisp. Example usage: display "Hello World!\n"then hit enter thrice. Voilà, you have wisp at the REPL!Caveeat: the wisp-parser is still experimental and contains known bugs. Use it for testing, but please do not rely on it for important stuff, yet.Update (2013-09-10): wisp-guile.w can now parse itself! Bootstrapping: The magical feeling of seeing a language (dialect) grow up to live by itself: python3 wisp.py wisp-guile.w > 1 && guile 1 wisp-guile.w > 2 && guile 2 wisp-guile.w > 3 && diff 2 3. Starting today, wisp is implemented in wisp.Update (2013-08-08): Wisp 0.3.1 released (Changelog).

Use Node:

⚙ Babcom is trying to load the comments ⚙

This textbox will disappear when the comments have been loaded.

If the box below shows an error-page, you need to install Freenet with the Sone-Plugin or set the node-path to your freenet node and click the Reload Comments button (or return).

If you see something like Invalid key: java.net.MalformedURLException: There is no @ in that URI! (Sone/search.html), you need to setup Sone and the Web of Trust

If you had Javascript enabled, you would see comments for this page instead of the Sone page of the sites author.

Note: To make a comment which isn’t a reply visible to others here, include a link to this site somewhere in the text of your comment. It will then show up here. To ensure that I get notified of your comment, also include my Sone-ID.

Link to this site and my Sone ID: sone://6~ZDYdvAgMoUfG6M5Kwi7SQqyS-gTcyFeaNN1Pf3FvY

This spam-resistant comment-field is made with babcom.

Kommentare

Darstellungsoptionen

Wählen Sie hier Ihre bevorzugte Anzeigeart für Kommentare und klicken Sie auf „Einstellungen speichern“ um die Änderungen zu übernehmen.

Sylvain Benner

Sylvain Benner gave constructive criticism on wisp which led to an interesting discussion and explanations of the reasons behind the design choices of wisp. He allowed me to share it here, too.

Sylvain Benner - 26.03.2013

I find it less readable and actually it adds complexity to the syntax because you have to deal with 3 components to understand the structure:

  • the new lines
  • the colon
  • the use of parentheses (in case of a multiple lines sexp due to line length)

So in fact you add 2 more constructs for readability which IMHO has not the expected effect on most people. Python is more readable because it simplify the constructs over C and alike, the thing is lisp is already dead simple syntax. Finally lisp is about lists and the python syntax just make it feel less closer this.

Arne Babenhauserheide - 26.03.2013

The main problem I see with lisp syntax for newcomers is that the most important character for recognizing an expression in text is the first character¹. And in lisp that is always the same: (

Also familiarity has a big impact on readability. And from my understanding, familiarity means a letter distribution which is similar to normal text.

Lisp actually is the best language when it comes to that: the basic constructs .,'` are the most common non-ascii-characters in normal text. But sadly the most common symmetric characters () are still very rare in normal text when compared to lisp. And those are in the most prominent place in Lisp: At the beginning of almost every line. And that makes lisp code look alien.

Those two reasons are why I kept working on indentation-sensitive syntax after taking the time to actually think it through for myself - and trying to find out why Python seemed so familiar from the start.

¹: I’m sure you can find linguistic research for that. I can offer an anecdote: I’m a roleplayer which means that I often invent and portray different people. From my experience there is one sure way to make the others mix up those different people: Give them names with the same first letter.

Sylvain Benner - 26.03.2013

Oh you can use the dot for line continuation :) Your syntax is a smart exercise and congrats for this achievement but it is not a good thing to use in the long run, learning a lisp with this syntax for new comers is not the best idea. 

Arne Babenhauserheide - 26.03.2013

I disagree: I obviously think that it’s a good thing if people use indentation based lisp, why else should I write the preprocessor? :)

You might need brackets if you want to reduce the indentation in continuation because that could actually enhance readability in some situations. This might be a fringe case (which is why I don’t mind if it’s a bit ugly), but it might crop up.

My main question is: Does the syntax manage to capture all features of lisp - is it a complete representation?

If that is true, then you should have all the power of lisp and the representation would be a purely aesthetic aspect (which is important, because it affects how the code feels, but it would then not be a limiting factor).

Sylvain Benner - 26.03.2013

Well, I gave it a quick try and 2 feedbacks:

  • the colon does not feel right for me, I would prefer a inlined colon and 2 levels of indentation beneath for the variable bindings list. In case of 1 list you would still be able to put it inline.

  • with a syntax highlighter it would be cool to colorize or bold the function calls. Having electric indentation would be cool. Maybe it is possible to achieve paredit by wrapping it (transform to lisp, apply paredit, transform to wisp).

Arne Babenhauserheide - 26.03.2013

Thanks for testing!

How would the inline colon look in code? (example)

syntax highlighting would really be cool, but before going that way, the syntax itself has to be right.

Sylvain Benner - 03:24

do you think it would make sens to drop the colon and keep only indentations to define the nested level, so only the following construct would be accepted ?

let
    var1 value1
    var2 value2
  dostuff var1 var2

Arne Babenhauserheide - 09:08

The colon cannot be dropped completely, because it is required when you have empty brackets or two blocks with double brackets. Use a simple defun or imagine a double let without action¹ (I’m sure you’ll find real code which fits better for this example):

(defun foo ()
  bar)

(doublelet
  ((foo bar))
  ((bla foo)))

The wisp version of this is

defun foo
  :
  . bar

doublelet
  :
    foo bar
  : ; <- this double backstep is the real issue
    bla foo

or shorter with inline colon (which you can only use if there are no not-tail-called functions inside the assignments).

defun foo :
  . bar

doublelet
  : foo bar
  : bla foo

The need to be able to represent things like this is the real reason, why the colon exists. The inline and start-of-line use is only a generalization of that principle (we add that syntax, so we should see how far we can push it to reduce the effective cost of introducing the additional syntax).

¹: I used a double let without action, even though that does nothing, because that makes it impossible to use later indentation. Another reason why I would not use later indentation to define whether something earlier is a single or double indent is that this would call for subtle and really hard to find errors:

defun flubb

    nubb
   gam

would become

(defun flubb ()
   ((nubb))
  (gam))

Also I think that fixed indentation width (alternative option to inferring it from later lines) would make it really hard to write readable code. Stuff like this would not be possible:

if
    equals wrong
           isright? stuff
    dostuff

Arne Babenhauserheide - 09:10

As syntax wisp has to be able to represent every lisp construct you can think of (to avoid being just a leaky abstraction where the writers still have to use brackets at times). Not every fringe case has to be beautiful, but it must be possible to represent it.

Arne Babenhauserheide - 19:42

Can I copy our discussion here into the comment section of the article (under GPL)? I think it could interest others, too.

Sylvain Benner - 03:36

Yes you can.

Arne Babenhauserheide - 10:03

Thanks!