This post contains a gentle introduction to procedural macros in Rust and a guide to writing a procedural macro to curry Rust functions. The source code for the entire library can be found here. It is also available on crates.io.

The following links might prove to be useful before getting started:

• Procedural Macros
• Currying

Or you can pretend you read them, because I have included a primer here :)

Contents

1. Currying
   
2. Procedural Macros
   
3. Definitions
   
4. Refinement
   
5. The In-betweens
        5.1 Dependencies
        5.2 The attribute macro
        5.3 Function Body
        5.4 Function Signature
        5.5 Getting it together
   
6. Debugging and Testing
   
7. Notes
   
8. Conclusion

Currying

Currying is the process of transformation of a function call like f(a, b, c) to f(a)(b)(c). A curried function returns a concrete value only when it receives all its arguments! If it does recieve an insufficient amount of arguments, say 1 of 3, it returns a curried function, that returns after receiving 2 arguments.

curry(f(a, b, c)) = h(a)(b)(c)

h(x) = g   <- curried function that takes upto 2 args (g)
g(y) = k   <- curried function that takes upto 1 arg (k)
k(z) = v   <- a value (v)

Keen readers will conclude the following,
h(x)(y)(z) = g(y)(z) = k(z) = v

Mathematically, if f is a function that takes two arguments x and y, such that x ϵ X, and y ϵ Y , we write it as:

f: (X × Y) -> Z

where × denotes the Cartesian product of set X and Y, and curried f (denoted by h here) is written as:

h: X -> (Y -> Z)

Procedural Macros

These are functions that take code as input and spit out modified code as output. Powerful stuff. Rust has three kinds of proc-macros:

• Function like macros
  
• Derive macros: #[derive(...)], used to automatically implement traits for structs/enums
  
• and Attribute macros: #[test], usually slapped onto functions

We will be using Attribute macros to convert a Rust function into a curried Rust function, which we should be able to call via: function(arg1)(arg2).

Definitions

Being respectable programmers, we define the input to and the output from our proc-macro. Here’s a good non-trivial function to start out with:

fn add(x: u32, y: u32, z: u32) -> u32 {
  return x + y + z;
}

Hmm, what would our output look like? What should our proc-macro generate ideally? Well, if we understood currying correctly, we should accept an argument and return a function that accepts an argument and returns … you get the point. Something like this should do:

fn add_curried1(x: u32) -> ? {
  return fn add_curried2 (y: u32) -> ? {
    return fn add_curried3 (z: u32) -> u32 {
      return x + y + z;
    }
  }
}

A couple of things to note:

Return types
We have placed ?s in place of return types. Let’s try to fix that. add_curried3 returns the ‘value’, so u32 is accurate. add_curried2 returns add_curried3. What is the type of add_curried3? It is a function that takes in a u32 and returns a u32. So a fn(u32) -> u32 will do right? No, I’ll explain why in the next point, but for now, we will make use of the Fn trait, our return type is impl Fn(u32) -> u32. This basically tells the compiler that we will be returning something function-like, a.k.a, behaves like a Fn. Cool!

If you have been following along, you should be able to tell that the return type of add_curried1 is:

impl Fn(u32) -> (impl Fn(u32) -> u32)

We can drop the parentheses because -> is right associative:

impl Fn(u32) -> impl Fn(u32) -> u32

Accessing environment
A function cannot access it’s environment. Our solution will not work. add_curried3 attempts to access x, which is not allowed! A closure¹ however, can. If we are returning a closure, our return type must be impl Fn, and not fn. The difference between the Fn trait and function pointers is beyond the scope of this post.

Refinement

Armed with knowledge, we refine our expected output, this time, employing closures:

fn add(x: u32) -> impl Fn(u32) -> impl Fn(u32) -> u32 {
  return move |y| move |z| x + y + z;
}

Alas, that does not compile either! It errors out with the following message:

error[E0562]: `impl Trait` not allowed outside of function
and inherent method return types
  --> src/main.rs:17:37
   |
   | fn add(x: u32) -> impl Fn(u32) -> impl Fn(u32) -> u32
   |                                   ^^^^^^^^^^^^^^^^^^^

You are allowed to return an impl Fn only inside a function. We are currently returning it from another return! Or at least, that was the most I could make out of the error message.

We are going to have to cheat a bit to fix this issue; with type aliases and a convenient nightly feature ²:

#![feature(type_alias_impl_trait)]  // allows us to use `impl Fn` in type aliases!

type T0 = u32;                 // the return value when zero args are to be applied
type T1 = impl Fn(u32) -> T0;  // the return value when one arg is to be applied
type T2 = impl Fn(u32) -> T1;  // the return value when two args are to be applied

fn add(x: u32) -> T2 {
  return move |y| move |z| x + y + z;
}

Drop that into a cargo project, call add(4)(5)(6), cross your fingers, and run cargo +nightly run. You should see a 15 unless you forgot to print it!

The In-Betweens

Let us write the magical bits that take us from function to curried function.

Initialize your workspace with cargo new --lib currying. Proc-macro crates are libraries with exactly one export, the macro itself. Add a tests directory to your crate root. Your directory should look something like this:

.
├── Cargo.toml
├── src
│   └── lib.rs
└── tests
    └── smoke.rs

Dependencies

We will be using a total of 3 external crates:

• proc_macro2
• syn
• quote

Here’s a sample Cargo.toml:

# Cargo.toml

[dependencies]
proc-macro2 = "1.0.9"
quote = "1.0"

[dependencies.syn]
version = "1.0"
features = ["full"]

[lib]
proc-macro = true  # this is important!

We will be using an external proc-macro2 crate as well as an internal proc-macro crate. Not confusing at all!

The attribute macro

Drop this into src/lib.rs, to get the ball rolling.

// src/lib.rs

use proc_macro::TokenStream;  // 1
use quote::quote;
use syn::{parse_macro_input, ItemFn};

#[proc_macro_attribute]   // 2
pub fn curry(_attr: TokenStream, item: TokenStream) -> TokenStream {
  let parsed = parse_macro_input!(item as ItemFn);  // 3
  generate_curry(parsed).into()  // 4
}

fn generate_curry(parsed: ItemFn) -> proc_macro2::TokenStream {}

1. Imports

A Tokenstream holds (hopefully valid) Rust code, this is the type of our input and output. Note that we are importing this type from proc_macro and not proc_macro2.

quote! from the quote crate is a macro that allows us to quickly produce TokenStreams. Much like the LISP quote procedure, you can use the quote! macro for symbolic transformations.

ItemFn from the syn crate holds the parsed TokenStream of a Rust function. parse_macro_input! is a helper macro provided by syn.

2. The lone export

Annotate the only pub of our crate with #[proc_macro_attribute]. This tells rustc that curry is a procedural macro, and allows us to use it as #[crate_name::curry] in other crates. Note the signature of the curry function. _attr is the TokenStream representing the attribute itself, item refers to the thing we slapped our macro into, in this case a function (like add). The return value is a modified TokenStream, this will contain our curried version of add.

3. The helper macro

A TokenStream is a little hard to work with, which is why we have the syn crate, which provides types to represent Rust tokens. An RArrow struct to represent the return arrow on a function and so on. One of those types is ItemFn, that represents an entire Rust function. The parse_macro_input! automatically puts the input to our macro into an ItemFn. What a gentleman!

4. Returning TokenStreams

We haven’t filled in generate_curry yet, but we can see that it returns a proc_macro2::TokenStream and not a proc_macro::TokenStream, so drop a .into() to convert it.

Lets move on, and fill in generate_curry, I would suggest keeping the documentation for syn::ItemFn and syn::Signature open.

// src/lib.rs

fn generate_curry(parsed: ItemFn) -> proc_macro2::TokenStream {
  let fn_body = parsed.block;      // function body
  let sig = parsed.sig;            // function signature
  let vis = parsed.vis;            // visibility, pub or not
  let fn_name = sig.ident;         // function name/identifier
  let fn_args = sig.inputs;        // comma separated args
  let fn_return_type = sig.output; // return type
}

We are simply extracting the bits of the function, we will be reusing the original function’s visibility and name. Take a look at what syn::Signature can tell us about a function:

                       .-- syn::Ident (ident)
                      /
                 fn add(x: u32, y: u32) -> u32
  (fn_token)      /     ~~~~~~~,~~~~~~  ~~~~~~
syn::token::Fn --'            /               \       (output)
                             '                 `- syn::ReturnType
             Punctuated<FnArg, Comma> (inputs)

Enough analysis, lets produce our first bit of Rust code.

Function Body

Recall that the body of a curried add should look like this:

return move |y| move |z| x + y + z;

And in general:

return move |arg2| move |arg3| ... |argN| <function body here>

We already have the function’s body, provided by fn_body, in our generate_curry function. All that’s left to add is the move |arg2| move |arg3| ... stuff, for which we need to extract the argument identifiers (doc: Punctuated, FnArg, PatType):

// src/lib.rs
use syn::punctuated::Punctuated;
use syn::{parse_macro_input, FnArg, Pat, ItemFn, Block};

fn extract_arg_idents(fn_args: Punctuated<FnArg, syn::token::Comma>) -> Vec<Box<Pat>> { 
  return fn_args.into_iter().map(extract_arg_pat).collect::<Vec<_>>();
}

Alright, so we are iterating over function args (Punctuated is a collection that you can iterate over) and mapping an extract_arg_pat to every item. What’s extract_arg_pat?

// src/lib.rs

fn extract_arg_pat(a: FnArg) -> Box<Pat> {
  match a {
    FnArg::Typed(p) => p.pat,
    _ => panic!("Not supported on types with `self`!"),
  }
}

FnArg is an enum type as you might have guessed. The Typed variant encompasses args that are written as name: type and the other variant, Reciever refers to self types. Ignore those for now, keep it simple.

Every FnArg::Typed value contains a pat, which is in essence, the name of the argument. The type of the arg is accessible via p.ty (we will be using this later).

With that done, we should be able to write the codegen for the function body:

// src/lib.rs

fn generate_body(fn_args: &[Box<Pat>], body: Box<Block>) -> proc_macro2::TokenStream {
  quote! {
    return #( move |#fn_args|  )* #body
  }
}

That is some scary looking syntax! Allow me to explain. The quote!{ ... } returns a proc_macro2::TokenStream, if we wrote quote!{ let x = 1 + 2; }, it wouldn’t create a new variable x with value 3, it would literally produce a stream of tokens with that expression.

The # enables variable interpolation. #body will look for body in the current scope, take its value, and insert it in the returned TokenStream. Kinda like quasi quoting in LISPs, you have written one.

What about #( move |#fn_args| )*? That is repetition. quote iterates through fn_args, and drops a move behind each one, it then places pipes (|), around it.

Let us test our first bit of codegen! Modify generate_curry like so:

// src/lib.rs

 fn generate_curry(parsed: ItemFn) -> TokenStream {
   let fn_body = parsed.block;
   let sig = parsed.sig;
   let vis = parsed.vis;
   let fn_name = sig.ident;
   let fn_args = sig.inputs;
   let fn_return_type = sig.output;

+  let arg_idents = extract_arg_idents(fn_args.clone());
+  let first_ident = &arg_idents.first().unwrap();

+  // remember, our curried body starts with the second argument!
+  let curried_body = generate_body(&arg_idents[1..], fn_body.clone());
+  println!("{}", curried_body);

   return TokenStream::new();
 }

Add a little test to tests/:

// tests/smoke.rs

#[currying::curry]
fn add(x: u32, y: u32, z: u32) -> u32 {
  x + y + z
}

#[test]
fn works() {
  assert!(true);
}

You should find something like this in the output of cargo test:

return move | y | move | z | { x + y + z }

Glorious println! debugging!

Function signature

This section gets into the more complicated bits of the macro, generating type aliases and the function signature. By the end of this section, we should have a full working auto-currying macro!

Recall what our generated type aliases should look like, for our add function:

type T0 = u32;
type T1 = impl Fn(u32) -> T0;
type T2 = impl Fn(u32) -> T1;

In general:

type T0 = <return type>;
type T1 = impl Fn(<type of arg N>) -> T0;
type T2 = impl Fn(<type of arg N - 1>) -> T1;
.
.
.
type T(N-1) = impl Fn(<type of arg 2>) -> T(N-2);

To codegen that, we need the types of:

• all our inputs (arguments)
• the output (the return type)

To fetch the types of all our inputs, we can simply reuse the bits we wrote to fetch the names of all our inputs! (doc: Type)

// src/lib.rs

use syn::{parse_macro_input, Block, FnArg, ItemFn, Pat, ReturnType, Type};

fn extract_type(a: FnArg) -> Box<Type> {
  match a {
    FnArg::Typed(p) => p.ty,  // notice `ty` instead of `pat`
      _ => panic!("Not supported on types with `self`!"),
  }
}

fn extract_arg_types(fn_args: Punctuated<FnArg, syn::token::Comma>) -> Vec<Box<Type>> {
  return fn_args.into_iter().map(extract_type).collect::<Vec<_>>();

}

A good reader would have looked at the docs for output member of the syn::Signature struct. It has the type syn::ReturnType. So there is no extraction to do here right? There are actually a couple of things we have to ensure here:

1. We need to ensure that the function returns! A function that does not return is pointless in this case, and I will tell you why, in the Notes section.

2. A ReturnType encloses the arrow of the return as well, we need to get rid of that. Recall:

   type T0 = u32
   // and not
   type T0 = -> u32

Here is the snippet that handles extraction of the return type (doc: syn::ReturnType):

// src/lib.rs

fn extract_return_type(a: ReturnType) -> Box<Type> {
  match a {
    ReturnType::Type(_, p) => p,
    _ => panic!("Not supported on functions without return types!"),
  }
}

You might notice that we are making extensive use of the panic! macro. Well, that is because it is a good idea to quit on receiving an unsatisfactory TokenStream.

With all our types ready, we can get on with generating type aliases:

// src/lib.rs

use quote::{quote, format_ident};

fn generate_type_aliases(
  fn_arg_types: &[Box<Type>],
  fn_return_type: Box<Type>,
  fn_name: &syn::Ident,
) -> Vec<proc_macro2::TokenStream> {    // 1

  let type_t0 = format_ident!("_{}_T0", fn_name);    // 2
  let mut type_aliases = vec![quote! { type #type_t0 = #fn_return_type  }];

  // 3
  for (i, t) in (1..).zip(fn_arg_types.into_iter().rev()) {
    let p = format_ident!("_{}_{}", fn_name, format!("T{}", i - 1));
    let n = format_ident!("_{}_{}", fn_name, format!("T{}", i));

    type_aliases.push(quote! {
        type #n = impl Fn(#t) -> #p
    });
  }

  return type_aliases;
}

1. The return value
We are returning a Vec<proc_macro2::TokenStream>, i. e., a list of TokenStreams, where each item is a type alias.

2. Format identifier?
I’ve got some explanation to do on this line. Clearly, we are trying to write the first type alias, and initialize our TokenStream vector with T0, because it is different from the others:

type T0 = something
// the others are of the form
type Tr = impl Fn(something) -> something

format_ident! is similar to format!. Instead of returning a formatted string, it returns a syn::Ident. Therefore, type_t0 is actually an identifier for, in the case of our add function, _add_T0. Why is this formatting important? Namespacing.

Picture this, we have two functions, add and subtract, that we wish to curry with our macro:

#[curry]
fn add(...) -> u32 { ... }

#[curry]
fn sub(...) -> u32 { ... }

Here is the same but with macros expanded:

type T0 = u32;
type T1 = impl Fn(u32) -> T0;
fn add( ... ) -> T1 { ... }

type T0 = u32;
type T1 = impl Fn(u32) -> T0;
fn sub( ... ) -> T1 { ... }

We end up with two definitions of T0! Now, if we do the little format_ident! dance we did up there:

type _add_T0 = u32;
type _add_T1 = impl Fn(u32) -> _add_T0;
fn add( ... ) -> _add_T1 { ... }

type _sub_T0 = u32;
type _sub_T1 = impl Fn(u32) -> _sub_T0;
fn sub( ... ) -> _sub_T1 { ... }

Voilà! The type aliases don’t tread on each other. Remember to import format_ident from the quote crate.

3. The TokenStream Vector

We iterate over our types in reverse order (T0 is the last return, T1 is the second last, so on), assign a number to each iteration with zip, generate type names with format_ident, push a TokenStream with the help of quote and variable interpolation.

If you are wondering why we used (1..).zip() instead of .enumerate(), it’s because we wanted to start counting from 1 instead of 0 (we are already done with T0!).

Getting it together

I promised we’d have a fully working macro by the end of last section. I lied, we have to tie everything together in our generate_curry function:

// src/lib.rs

 fn generate_curry(parsed: ItemFn) -> proc_macro2::TokenStream {
   let fn_body = parsed.block;
   let sig = parsed.sig;
   let vis = parsed.vis;
   let fn_name = sig.ident;
   let fn_args = sig.inputs;
   let fn_return_type = sig.output;

   let arg_idents = extract_arg_idents(fn_args.clone());
   let first_ident = &arg_idents.first().unwrap();
   let curried_body = generate_body(&arg_idents[1..], fn_body.clone());

+  let arg_types = extract_arg_types(fn_args.clone());
+  let first_type = &arg_types.first().unwrap();
+  let type_aliases = generate_type_aliases(
+      &arg_types[1..],
+      extract_return_type(fn_return_type),
+      &fn_name,
+  );

+  let return_type = format_ident!("_{}_{}", &fn_name, format!("T{}", type_aliases.len() - 1));

+  return quote! {
+      #(#type_aliases);* ;
+      #vis fn #fn_name (#first_ident: #first_type) -> #return_type {
+          #curried_body ;
+      }
+  };
 }

Most of the additions are self explanatory, I’ll go through the return statement with you. We are returning a quote!{ ... }, so a proc_macro2::TokenStream. We are iterating through the type_aliases variable, which you might recall, is a Vec<TokenStream>. You might notice the sneaky semicolon before the *. This basically tells quote, to insert an item, then a semicolon, and then the next one, another semicolon, and so on. The semicolon is a separator. We need to manually insert another semicolon at the end of it all, quote doesn’t insert a separator at the end of the iteration.

We retain the visibility and name of our original function. Our curried function takes as args, just the first argument of our original function. The return type of our curried function is actually, the last type alias we create. If you think back to our manually curried add function, we returned T2, which was in fact, the last type alias we created.

I am sure, at this point, you are itching to test this out, but before that, let me introduce you to some good methods of debugging proc-macro code.

Debugging and Testing

Install cargo-expand via:

cargo install cargo-expand

cargo-expand is a neat little tool that expands your macro in places where it is used, and lets you view the generated code! For example:

# create a bin package hello
$ cargo new hello

# view the expansion of the println! macro
$ cargo expand

#![feature(prelude_import)]
#[prelude_import]
use std::prelude::v1::*;
#[macro_use]
extern crate std;
fn main() {
  {
    ::std::io::_print(::core::fmt::Arguments::new_v1(
        &["Hello, world!\n"],
        &match () {
            () => [],
        },
      ));
  };
}

Writing proc-macros without cargo-expand is tantamount to driving a vehicle without rear view mirrors! Keep an eye on what is going on behind your back.

Now, your macro won’t always compile, you might just recieve the bee movie script as an error. cargo-expand will not work in such cases. I would suggest printing out your variables to inspect them. TokenStream implements Display as well as Debug. We don’t always have to be respectable programmers. Just print it.

Enough of that, lets get testing:

// tests/smoke.rs

#![feature(type_alias_impl_trait)]

#[crate_name::curry]
fn add(x: u32, y: u32, z: u32) -> u32 {
   x + y + z
}

#[test]
fn works() {
  assert_eq!(15, add(4)(5)(6));
}

Run cargo +nightly test. You should see a pleasing message:

running 1 test
test tests::works ... ok

Take a look at the expansion for our curry macro, via cargo +nightly expand --tests smoke:

type _add_T0 = u32;
type _add_T1 = impl Fn(u32) -> _add_T0;
type _add_T2 = impl Fn(u32) -> _add_T1;
fn add(x: u32) -> _add_T2 {
  return (move |y| {
    move |z| {
      return x + y + z;
    }
  });
}

// a bunch of other stuff generated by #[test] and assert_eq!

A sight for sore eyes.

Here is a more complex example that generates ten multiples of the first ten natural numbers:

#[curry]
fn product(x: u32, y: u32) -> u32 {
  x * y
}

fn multiples() -> Vec<Vec<u32>>{
  let v = (1..=10).map(product);
  return (1..=10)
      .map(|x| v.clone().map(|f| f(x)).collect())
      .collect();
}

Notes

I didn’t quite explain why we use move |arg| in our closure. This is because we want to take ownership of the variable supplied to us. Take a look at this example:

let v = add(5);
let g;
{
  let x = 5;
  g = v(x);
}
println!("{}", g(2));

Variable x goes out of scope before g can return a concrete value. If we take ownership of x by moveing it into our closure, we can expect this to work reliably. In fact, rustc understands this, and forces you to use move.

This usage of move is exactly why a curried function without a return is useless. Every variable we pass to our curried function gets moved into its local scope. Playing with these variables cannot cause a change outside this scope. Returning is our only method of interaction with anything beyond this function.

Conclusion

Currying may not seem to be all that useful. Curried functions are unwieldy in Rust because the standard library is not built around currying. If you enjoy the possibilities posed by currying, consider taking a look at Haskell or Scheme.

My original intention with peppe.rs was to post condensed articles, a micro blog, but this one turned out extra long.

Perhaps I should call it a ‘macro’ blog :)

I’ve always been an admirer of pixel art, because of it’s simplicity and it’s resemblance to bitmap font design. Recently, I decided to take the dive and make some art of my own.

I used GIMP because I am fairly familiar with it. Aseprite seems to be the editor of choice for animated pixel art though.

Setting up the canvas

Picking a canvas size is daunting. Too small, and you won’t be able to fit in enough detail to make a legible piece. Too big and you’ve got too many pixels to work with!

I would suggest starting out with anywhere between 100x100 and 200x200. Here’s a sample configuration.

Sometimes I use a 10x10 grid, View > Show Grid and Edit > Preferences > Default Grid > Spacing, but that can get jarring, so I throw down a couple of guides, drag right or down from the left or top gutters for vertical and horizontal guides respectively.

Choosing a Brush

The most important part of our setup is the brush. Use the Pencil Tool (n on the keyboard) for hard edge drawings. Here’s a small comparison if you don’t know the difference between a hard edge and a soft edge:

I turn the size down all the way to 1 ([ on the keyboard). Set Dynamics off. Here’s a sample brush configuration.

Laying down the pixels!

With the boring stuff out of the way, we can start with our piece. I usually follow a three step process:

• draw a rough outline
• fill in the shadows
• add highlights

But this process is better explained with an example: an onigiri. Let us start off with a 100x100 canvas.

Drawing the outline

For the most part, our figure will be symmetric. If you are on GIMP 2.10+, you can take advantage of the Symmetry Painting feature. Go ahead and enable vertical symmetry, Window > Dockable Dialogs > Symmetry Painting and Symmetry Painting > Symmetry > Mirror > Vertical.

If you are running an older version of GIMP, draw in the left side, duplicate the layer, flip it horizontally, and merge it with the original.

Your outline might look something like this:

Go ahead and fill it in with the fill tool (Shift + b on the keyboard), add in some seaweed as well, preferably on a different layer. You can toggle symmetry on and off to save yourself some time.

Shadows

For now, let us focus on the shadows on the object itself, we’ll come back to the shadows cast by the object on the surface later.

Shadows on any surface always follow the shape of the surface. A spherical onigiri would have a circular shadow:

A couple of noticeable changes:

Layers: The layer containing the seaweed has been hidden.
Color: The color of the shadow is just a slightly lighter version of the original object (reduce the Value on the HSV scale).
Area: The shadow does not go all the way (notice the bottom edges).

The shadow does not go all the way because we will be filling in that area with another, darker shadow! An image might explain better:

To emulate soft lights, reduce the value by 2 to 3 points every iteration. Notice how area 1 is much larger than area 4. This is because an onigiri resembles a bottom heavy oblate spheroid, a sphere that is slightly fatter around the lower bottom, and areas 1 and 2 catch more light than areas 3 and 4.

Do the same with the seaweed. The seaweed, being a smaller, flatter object, doesn’t cast much of a shadow, so stop with 1 or 2 iterations of the gradient:

We’re getting there!

Highlights

This step handles the details on the strongly illuminated portions of the object. Seaweed is a bit glossy, lighten the edges to make it seem shiny. The rice is not as shiny, but it does form an uneven surface. Add in some shadows to promote the idea of rice grains. Here is the finished result:

Finishing Touches

Some color correction and a e s t h e t i c Japanese text later, our piece is complete!

Hold on, why is it so tiny? Well, that’s because our canvas was 100x100, head over to Image > Scale Image, set Quality > Interpolation to None and scale it up to 700x700, et voilà!

Last weekend, I was tasked with refactoring the 96 unit tests on ruma-events to use strictly typed json objects using serde_json::json! instead of raw strings. It was rather painless thanks to vim :)

Here’s a small sample of what had to be done (note the lines prefixed with the arrow):

→ use serde_json::{from_str};
  
  #[test]
  fn deserialize() {
    assert_eq!(
→       from_str::<Action>(r#"{"set_tweak": "highlight"}"#),
        Action::SetTweak(Tweak::Highlight { value: true })
        );
  }

had to be converted to:

→ use serde_json::{from_value};
  
  #[test]
  fn deserialize() {
    assert_eq!(
→       from_value::<Action>(json!({"set_tweak": "highlight"})),
        Action::SetTweak(Tweak::Highlight { value: true })
        );
  }

The arglist

For the initial pass, I decided to handle imports, this was a simple find and replace operation, done to all the files containing tests. Luckily, modules (and therefore files) containing tests in Rust are annotated with the #[cfg(test)] attribute. I opened all such files:

# `grep -l pattern files` lists all the files
#  matching the pattern

vim $(grep -l 'cfg\(test\)' ./**/*.rs)

# expands to something like:
vim push_rules.rs room/member.rs key/verification/lib.rs

Starting vim with more than one file at the shell prompt populates the arglist. Hit :args to see the list of files currently ready to edit. The square [brackets] indicate the current file. Navigate through the arglist with :next and :prev. I use tpope’s vim-unimpaired ¹, which adds ]a and [a, mapped to :next and :prev.

All that’s left to do is the find and replace, for which we will be using vim’s argdo, applying a substitution to every file in the arglist:

:argdo s/from_str/from_value/g

The quickfix list

Next up, replacing r#" ... "# with json!( ... ). I couldn’t search and replace that trivially, so I went with a macro call ² instead, starting with the cursor on ‘r’, represented by the caret, in my attempt to breakdown the process:

BUFFER:    r#" ... "#;
           ^

ACTION:    vllsjson!(

BUFFER     json!( ... "#;
                ^

ACTION:    <esc>$F#

BUFFER:    json!( ... "#;
                       ^

ACTION:    vhs)<esc>

BUFFER:    json!( ... );

Here’s the recorded ³ macro in all its glory: vllsjson!(<esc>$F#vhs)<esc>.

Great! So now we just go ahead, find every occurrence of r# and apply the macro right? Unfortunately, there were more than a few occurrences of raw strings that had to stay raw strings. Enter, the quickfix list.

The idea behind the quickfix list is to jump from one position in a file to another (maybe in a different file), much like how the arglist lets you jump from one file to another.

One of the easiest ways to populate this list with a bunch of positions is to use vimgrep:

# basic usage
:vimgrep pattern files

# search for raw strings
:vimgrep 'r#' ./**/*.rs

Like :next and :prev, you can navigate the quickfix list with :cnext and :cprev. Every time you move up or down the list, vim indicates your index:

(1 of 131): r#"{"set_tweak": "highlight"}"#;

And just like argdo, you can cdo to apply commands to every match in the quickfix list:

:cdo norm! @q

But, I had to manually pick out matches, and it involved some button mashing.

External Filtering

Some code reviews later, I was asked to format all the json inside the json! macro. All you have to do is pass a visual selection through a pretty json printer. Select the range to be formatted in visual mode, and hit :, you will notice the command line displaying what seems to be gibberish:

:'<,'>

'< and '> are marks ⁴. More specifically, they are marks that vim sets automatically every time you make a visual selection, denoting the start and end of the selection.

A range is one or more line specifiers separated by a ,:

:1,7       lines 1 through 7
:32        just line 32
:.         the current line
:.,$       the current line to the last line
:'a,'b     mark 'a' to mark 'b'

Most : commands can be prefixed by ranges. :help usr_10.txt for more on that.

Alright, lets pass json through python -m json.tool, a json formatter that accepts stdin (note the use of ! to make use of an external program):

:'<,'>!python -m json.tool

Unfortunately that didn’t quite work for me because the range included some non-json text as well, a mix of regex and macros helped fix that. I think you get the drift.

Another fun filter I use from time to time is :!sort, to sort css attributes, or :!uniq to remove repeated imports.

I am not an expert with fonts, but I do have some experience ¹, and common sense. This post aims to debunk some misconceptions about font sizes!

11 px on your display is probably not 11 px on my display. Let’s do some quick math. I have two displays, 1366x768 @ 21" and another with 1920x1080 @ 13", call them A and B for now.

Display A has 1,049,088 pixels. A pixel is a square, of side say, s cm. The total area covered by my 21" display is about 1,066 cm^2 (41x26). Thus,

Display A
Dimensions: 1366x768 @ 21" (41x26 sq. cm)
1,049,088 s^2 = 1066
            s = 0.0318 cm (side of a pixel on Display A)

Bear with me, as I repeat the number crunching for Display B:

Display B
Dimensions: 1920x1080 @ 13" (29.5x16.5 sq. cm)
2,073,600 s^2 = 486.75
            s = 0.0153 cm (side of a pixel on Display B)

The width of a pixel on Display A is double the width of a pixel on Display B. The area occupied by a pixel on Display A is 4 times the area occupied by a pixel on Display B.

The size of a pixel varies from display to display!

A 5x11 bitmap font on Display A would be around 4 mm tall whereas the same bitmap font on Display B would be around 1.9 mm tall. A 11 px tall character on B is visually equivalent to a 5 px character on A. When you view a screenshot of Display A on Display B, the contents are shrunk down by a factor of 2!

So screen resolution is not enough, how else do we measure size? Pixel Density! Keen readers will realize that the 5^th grade math problem we solved up there showcases pixel density, or, pixels per cm (PPCM). Usually we deal with pixels per inch (PPI).

Note: PPI is not to be confused with DPI ² (dots per inch). DPI is defined for printers.

In our example, A is a 75 ppi display and B is around 165 ppi ³. A low ppi display appears to be ‘pixelated’, because the pixels are more prominent, much like Display A. A higher ppi usually means you can view larger images and render crispier fonts. The average desktop display can stuff 100-200 pixels per inch. Smart phones usually fall into the 400-600 ppi (XXXHDPI) category. The human eye fails to differentiate detail past 300 ppi.

So … streaming an 8K video on a 60" TV provides the same clarity as a HD video on a smart phone?

Absolutely. Well, clarity is subjective, but the amount of detail you can discern on mobile displays has always been limited. Salty consumers of the Xperia 1 ⁴ will say otherwise.

Maybe I will talk about font rendering in another post, but thats all for now. Don’t judge a font size by its screenshot.

I learnt about termux from a friend on IRC recently. It looked super gimmicky to me at first, but it eventually proved to be useful. Here’s what I use it for:

rsync

Ever since I degoogled my android device, syncing files between my phone and my PC has always been a pain. I’m looking at you MTP. But, with termux and sshd all set up, it’s as simple as:

$ arp
Address         HWtype  HWad ...
192.168.43.187  ether   d0:0 ...

$ rsync -avz 192.168.43.187:~/frogs ~/pics/frogs

ssh & tmux

My phone doubles as a secondary view into my main machine with ssh and tmux. When I am away from my PC (read: sitting across the room), I check build status and IRC messages by sshing into a tmux session running the said build or weechat.

file uploads

Not being able to access my (ssh-only) file host was crippling. With a bash instance on my phone, I just copied over my ssh keys, and popped in a file upload script (a glorified scp). Now I just have to figure out a way to clean up these file names …

~/storage/pictures/ $ ls
02muf5g7b2i41.jpg  7alt3cwg77841.jpg  cl4bsrge7id11.png
mtZabXG.jpg        p8d5c584f2841.jpg  vjUxGjq.jpg

cmus

Alright, I don’t really listen to music via cmus, but I did use it a couple times when my default music player was acting up. cmus is a viable option:

My 4th semester involves ARM programming. And proprietary tooling (Keil C). But we don’t do that here.

Building

Assembling and linking ARM binaries on non-ARM architecture devices is fairly trivial. I went along with the GNU cross bare metal toolchain binutils, which provides arm-as and arm-ld (among a bunch of other utils that I don’t care about for now).

Assemble .s files with:

arm-none-eabi-as main.s -g -march=armv8.1-a -o main.out

The -g flag generates extra debugging information that gdb picks up. The -march option establishes target architecture.

Link .o files with:

arm-none-eabi-ld main.out -o main

Running (and Debugging)

Things get interesting here. gdb on your x86 machine cannot read nor execute binaries compiled for ARM. So, we simulate an ARM processor using qemu. Now qemu allows you to run gdbserver on startup. Connecting our local gdb instance to gdbserver gives us a view into the program’s execution. Easy!

Run qemu, with gdbserver on port 1234, with our ARM binary, main:

qemu-arm -singlestep -g 1234 main

Start up gdb on your machine, and connect to qemu’s gdbserver:

(gdb) set architecture armv8-a
(gdb) target remote localhost:1234
(gdb) file main
Reading symbols from main...  # yay!

GDB Enhanced

gdb is cool, but it’s not nearly as comfortable as well fleshed out emulators/IDEs like Keil. Watching registers, CPSR and memory chunks update is pretty fun.

I came across gdb’s TUI mode (hit C-x C-a or type tui enable at the prompt). TUI mode is a godsend. It highlights the current line of execution, shows you disassembly outputs, updated registers, active breakpoints and more.

But, it is an absolute eyesore.

Say hello to GEF! “GDB Enhanced Features” teaches our old dog some cool new tricks. Here are some additions that made my ARM debugging experience loads better:

• Memory watches
• Register watches, with up to 7 levels of deref (overkill, I agree)
• Stack tracing

And it’s pretty! See for yourself:

Editing

Vim, with syntax off because it dosen’t handle GNU ARM syntax too well.

This piece aims to highlight (pun intended) some of the reasons behind my color free editor setup.

Imagine highlighting an entire book because all of it is important. That is exactly what (most) syntax highlighting does. It is difficult for the human eye to filter out noise in rainbow barf. Use color to draw attention, not diverge it.

At the same time, a book devoid of color is boring! What is the takeaway from this 10 line paragraph? What are the technical terms used?

Prose and code are certainly different, but the fickle minded human eye is the same. The eye constantly looks for a frame of reference, a focal point. It grows tired when it can’t find one.

The following comparison does a better job of explaining (none, ample and over-the-top highlighting, from left to right):

Without highlighting (far left), it is hard to differentiate between comments and code! The florid color scheme (far right) is no good either, it contains too many attention grabbers. The center sample is a healthy balance of both. Function calls and constants stand out, and repetitive keywords and other noise (let, as) are mildly dimmed out. Comments and non-code text (sign column, status text) are dimmed further.

I’ll stop myself before I rant about color contrast and combinations.

After going through a bunch of static site generators (pelican, hugo, vite), I decided to roll my own. If you are more of the ‘show me the code’ kinda guy, here you go.

Text formatting

I chose to write in markdown, and convert to html with lowdown.

Directory structure

I host my site on GitHub pages, so docs/ has to be the entry point. Markdown formatted posts go into posts/, get converted into html, and end up in docs/index.html, something like this:

posts=$(ls -t ./posts)     # chronological order!
for f in $posts; do
    file="./posts/"$f      # `ls` mangled our file paths
    echo "generating post $file"

    html=$(lowdown "$file")
    echo -e "html" >> docs/index.html
done

Assets

Most static site generators recommend dropping image assets into the site source itself. That does have it’s merits, but I prefer hosting images separately:

# strip file extension
ext="${1##*.}"

# generate a random file name
id=$( cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 2 | head -n 1 )
id="$id.$ext"

# copy to my file host
scp -P 443 "$1" emerald:files/"$id" 
echo "https://u.peppe.rs/$id"

Templating

generate.sh brings the above bits and pieces together (with some extra cruft to avoid javascript). It uses sed to produce nice titles from the file names (removes underscores, title-case), and date(1) to add the date to each post listing!

Decided to do one of these because everyone does one of these.

My entire setup is managed with GNU stow, making it easier to replicate on fresh installations. You can find my configuration files on GitHub.

I run Void Linux (glibc) on my HP Envy 13" (2018). To keep things simple, I run a raw X session with 2bwm as my window manager, along with dunst (notification daemon) and Sam’s compton (compositor) fork.

I am a fan of GNU tools, so I use bash as my shell, and coreutils to manage files, archives, strings, paths etc. I edit files with vim, chat with weechat, listen to music with cmus, monitor processes with htop, manage sessions with tmux, read pdfs in zathura. I rarely ever leave the comfort of my terminal emulator, urxvt.

Most of my academic typesetting is done with TeX, and compiled with xelatex. Other fun documents are made with GIMP :).

I finally got around to installing Void GNU/Linux on my main computer. Rolling release, non-systemd, need I say more?

As with all GNU/Linux distributions, wireless networks had me in a fix. If you can see this post, it means I’ve managed to get online. It turns out, wpa_supplicant was detecting the wrong interface by default (does it ever select the right one?). Let us fix that:

$ sudo rm -r /var/service/wpa_supplicant
$ sudo killall dhcpcd

What is the right interface though?

$ iw dev
   ...
   Interface wlp2s0
   ...

Aha! Let us run wpa_supplicant on that interface, as a background process:

$ sudo wpa_supplicant -B -i wlp2s0 -c /etc/wpa_supplicant/wpa_supplicant.conf
$ sudo dhcpcd -B wlp2s0
$ ping google.com
PING ...

Yay! Make those changes perpetual by enabling the service:

------------------------------------------------------
# Add these to /etc/wpa_supplicant/wpa_supplicant.conf
OPTS="-B"
WPA_INTERFACE="wlp2s0"
------------------------------------------------------
$ sudo ln -s /etc/sv/wpa_supplicant /var/service/
$ sudo ln -s /etc/sv/dhcpcd /var/service/
$ sudo sv restart wpa_supplicant
$ sudo sv restart dhcpcd

Glyph Bitmap Distribution Format is no more, as the creators of Pango, one of the most widely used text rendering libraries, announced their plans for Pango 1.44.

Until recently, Pango used FreeType to draw fonts. They will be moving over to Harfbuzz, an evolution of FreeType.

Why?

In short, FreeType was hard to work with. It required complex logic, and provided no advantage over Harfbuzz (other than being able to fetch opentype metrics with ease).

Upgrading to Pango v1.44 will break your GTK applications (if you use a bdf/pcf bitmap font). Harfbuzz does support bitmap-only OpenType fonts, otbs. Convert your existing fonts over to otbs using FontForge. It is to be noted that applications such as xterm and rxvt use xft (X FreeType) to render fonts, and will remain unaffected by the update.

Both scientifica and curie will soon ship with bitmap-only OpenType font formats.

Onivim is a ‘modern modal editor’, combining fancy interface and language features with vim-style modal editing. What’s wrong you ask?

Apart from buggy syntax highlighting, broken scrolling and others, Onivim is proprietary software. It is licensed under a commercial end user agreement license, which prohibits redistribution in both object code and source code formats.

Onivim’s core editor logic (bits that belong to vim), have been separated from the interface, into libvim. libvim is licensed under MIT, which means, this ‘extension’ of vim is perfectly in adherence to vim’s license text! Outrun Labs are exploiting this loophole (distributing vim as a library) to commercialize Onivim.

Onivim’s source code is available on GitHub. They do mention that the source code trickles down to the oni2-mit repository, which (not yet) contains MIT-licensed code, 18 months after each commit to the original repository.

Want to contribute to Onivim? Don’t. They make a profit out of your contributions. Currently, Onivim is priced at $19.99, ‘pre-alpha’ pricing which is 80% off the final price! If you are on the lookout for an editor, I would suggest using Vim, charity ware that actually works, and costs $100 lesser.

Bash is tricky, don’t let your editor get in your way. Here’s a couple of neat additions you could make to your vimrc for a better shell programming experience.

Man pages inside vim

Source this script to get started:

runtime ftplugin/man.vim

Now, you can open manpages inside vim with :Man! It adds nicer syntax highlighting and the ability to jump around with Ctrl-] and Ctrl-T.

By default, the manpage is opened in a horizontal split, I prefer using a new tab:

let g:ft_man_open_mode = 'tab'

Scratchpad to test your commands

I often test my sed substitutions, here is a sample from the script used to generate this site:

# a substitution to convert snake_case to Title Case With Spaces
echo "$1" | sed -E -e "s/\..+$//g"  -e "s/_(.)/ \u\1/g" -e "s/^(.)/\u\1/g"

Instead of dropping into a new shell, just test it out directly from vim!

• Yank the line into a register:

yy

• Paste it into the command-line window:

q:p

• Make edits as required:

syntax off            # previously run commands
edit index.html       # in a buffer!
w | so %
!echo "new_post.md" | sed -E -e "s/\..+$//g"  --snip--
^--- note the use of '!'

• Hit enter with the cursor on the line containing your command!

$ vim
New Post         # output
Press ENTER or type command to continue

Often times, when I run a vim command that makes “big” changes to a file (a macro or a :vimgrep command) I lose my original position and feel disoriented.

Save position with winsaveview()!

The winsaveview() command returns a Dictionary that contains information about the view of the current window. This includes the cursor line number, cursor coloumn, the top most line in the window and a couple of other values, none of which concern us.

Before running our command (one that jumps around the buffer, a lot), we save our view, and restore it once its done, with winrestview.

let view = winsaveview()
s/\s\+$//gc              " find and (confirm) replace trailing blanks
winrestview(view)        " restore our original view!

It might seem a little overkill in the above example, just use `` (double backticks) instead, but it comes in handy when you run your file through heavier filtering.

a couple of nifty tricks to help you copy-paste better:

1. reselecting previously selected text (i use this to fix botched selections):

   gv  " :h gv for more
       " you can use `o` in visual mode to go to the `Other` end of the selection
       " use a motion to fix the selection

2. reselecting previously yanked text:

   `[v`]
   `[         " marks the beginning of the previously yanked text   :h `[
   `]         " marks the end                                       :h `]
    v         " visual select everything in between
   
   nnoremap gb `[v`]    " "a quick map to perform the above

3. pasting and indenting text (in one go):

   ]p   " put (p) and adjust indent to current line
   ]P   " put the text before the cursor (P) and adjust indent to current line