A couple of side-project updates: keeping my archival tool alive, ssh-over-web-browser tailored to my needs, and a slightly better dictation CLI.

Archivist Updates

I continue to use, and get a lot of value out of, Archivist.

Since the original post about the project, six years ago (sic!), the tool got extended with collecting screenshots, finding similar items across all sources, and recently moved to my new Home Server, hosted as a permanent web app.

For collecting screenshots, I use a two-tier system:

1. I have a couple of Hammerspoon shortcuts overriding the default macOS ones, that take the screenshot, place it in Dropbox/Screenshots/ folder, and inject OCR text into the files through a combination of tinyocr and xattr. Dropbox only supports a subset of xattr, so this took some figuring-out, but I managed to find a way to sync them with the Home Server, with the OCR data surviving in the file. I could do OCR purely in Archivist and save some headaches, but the nice side-effect of having it in xattr is that Spotlight indexes that data, so I can do a quick search for image content directly from Spotlight.
2. Archivist does a refresh of all its sources hourly, and as part of that, picks up any changes to the Dropbox folder, and updates its local DB.

Similarity search was a fun recent addition. As part of the hourly refresh, all images are processed on-device with DINOv2-base for embeddings (after some extra post-processing to make them square, without alpha channel, etc.). The resulting vector is stored in the SQLite as a blob of numbers.

Together with this, textual values (like the OCR info, metadata, etc.) are embedded with MiniLM-L6-v2, and stored into another table.

The final search uses HNSW index for super fast fetches (under 16ms for my ~10k items), and combines results from image and text similarity searches.

All together, we get this:

Telegraphist

On the topic of always-on custom services, I couldn't find any app that would tick all the boxes for my ssh-from-iPhone needs:

• I continue to stubbornly use base16-shell for terminal theming, so I can very easily set colors with a simple command, even though I rarely use it.
• I wanted something that works with terminal graphics, as I — also rarely — use timg to poke at images inline.
• Finally, on the Home Server, I'm always in a tmux session, so a native support for that would be nice.

I poked around AppStore, but couldn't find anything ticking at least some of these boxes. I also tested a handful of "web TTY" kinds of things, but nothing sparked joy. So, I made my own TTY-in-a-browser: Telegraphist.

It didn't take much prompting to combine a PTY library, a WebSocket connection, and xterm.js and get something tailor-made for me.

The base16-shell colors not only work in the terminal, but also impact the minimal UI:

I can look at images:

And there's a native tmux integration:

The most fun part of this project was cutting corners to get to a rough first working version, and then working on improvements from within itself.

Tinyparakeet

Exactly a year ago I wrote about a small Whisper wrapper that I used to use for dictation.

Since then I discovered parakeet family of models, and made tinyparakeet — a small CLI tool for speech-to-text, fully on-device:

It's open source if you, like me, rarely have a need for dictating things, and mostly live in CLI: szymonkaliski/tinyparakeet.

Worth Checking Out

What I've been reading lately:

• The Charisma Machine, recommended by a friend, a great "Look for Opposite Views Instead of Confirming Ones" kind of book on Constructionism.
• Theory and Reality, a great meta-overview of philosophy of science themes. I was already familiar with most of the original works, but it was still fun to go through a refresher, and see them contrasted against each other.
• The Theoretical Minimum, the second half of which was a bit too dense for me right now. Physics has never been my strong suit, and I want to get a bit better at it.

On the web:

• a great talk from Andy Matuschak on LLMs and Malleable Software
  • and on that topic, Deadmau5 vibe-coding his own music production tool
• lovely morphing visualizations
• Bret Victor on "Time and Possibility" — the Dynamicland Archive remains a treasure trove
• visualizing p5.js execution
• "I'm not worried about the machines. The machines are fine. I'm worried about us."

A pretty nerdy update this time around — I replaced my Raspberry Pi 4 with a Mini PC running NixOS, I started using MicroVMs for sandboxing on said Mini PC, I have some thoughts about feedback loops for LLMs, and a couple of side-project updates.

If you missed the last issue, I'm an independent consultant now, currently exploring at Google Creative Lab.

I currently have some capacity for advising, and I'm always happy to discuss future projects!

Reach out if you're interested in working together: hi@szymonkaliski.com

Home Server on NixOS

I've written about my smart home setup before. Back then, the whole thing was controlled by a Raspberry Pi 4, which recently started to feel a bit constraining — on one hand I had an always-on server in my home, on the other, I couldn't really do much with it.

The issues started with a slowly degrading SD card — this isn't that big of a deal in itself — I could just clone my stuff onto a new one, but I haven't updated Raspbian since I flashed it originally, so by now I couldn't really install anything new.

At the same time I was afraid of how problematic it would be to set everything back up from scratch if I update. I also got somewhat spoiled by running Nix on my Macbook, it's just way too nice to go back to a normal way of doing things, and while I could put NixOS on my Raspberry Pi, it felt like asking for extra troubleshooting.

I was on the fence about upgrading for weeks, but the rising RAM prices (and conversations with a friend) pushed me over the edge. I got myself a N350 in a chunky passively-cooled aluminium case:

At first I thought this would be overkill, but once I started thinking about all the things I could do there, I was pretty happy I went with the beefy option.

NixOS installation was surprisingly easy. I went with the graphical installer booted from a USB stick, clicked through a couple of screens, selected "no thank you" for the final graphical environment, and was ready to go.

The first step of the setup was a fun bootstrapping problem. I already have my dotfiles online, with a simple script which — in theory — sets everything up, but then it turned out git is not in the NixOS shell by default. I ended up having to write this fun command:

nix --extra-experimental-features "nix-command flakes" \
  run nixpkgs#git -- clone https://github.com/szymonkaliski/home-configuration.git

...and then went through a cycle of running ./setup.sh, failing, fixing what's failing, and doing it again. This took a couple of attempts, but now I can be sure I can stand up a new machine in a single bash script, which makes me much less worried about upgrading the hardware at some point in the future.

The declarative and git-tracked configuration of the whole machine is really great. There's no need to sudoedit random files in /etc/, and note down (or fail to do so) what I did. Every change can be easily rolled back, and the system is, in a sense, stateless — which makes experimentation super "cheap" — I can try out a new service for a while, and if I don't like it, remove a couple of lines of nix and rebuild the system. LLMs are pretty good at writing these configs too, so most of my interaction with the machine is now like "hey LLM can you make sure this script runs every hour?"

MicroVMs

NixOS enables another cool trick — microvm.nix — a way to easily manage high-performance VMs (which are much better at isolation from the host OS than Docker).

My config creates four small ephemeral VMs with bare-bones user environment — just enough so direnv can automatically evaluate flake.nix, and I can safely --dangerously-skip-permissions. This allows me to let the Agents run free, even when my laptop is closed, and not worry about anything breaking.

I can also run the same project multiple times in parallel! There's no messing around with ports, setting up multiple DB instances, etc. — the same configuration with the same npm run dev, just in a full-on separate virtual machine.

I use a small microvm script which allows me to do things like cd some-project && microvm start which picks up the first free VM, mounts the current directory into /workspace, starts and ssh-es into the machine, and enters /workspace so I'm ready to go (in about 15s).

Thanks to Tailscale, I can open http://vm-1:3000 wherever I am, and that will get me not only into my Mini PC, but also into the VM running inside of it.

Feedback Loops for LLMs

The "Agents running free" leads me to a game I started playing recently: what kind of ad-hoc tools and feedback loops can I build for the task-at-hand, so the model has a very concrete thing to iterate against, easy visibility into its state, and a clear success criteria. Some examples below.

MirrorState has been effective at letting the models reason about the application state — any sort of complex Chrome/Playwright MCP calls are replaced with reading from a single JSON blob (this assumes you have the page open in a browser, so it's a more hands-on workflow). It takes a bit of AGENT.md massaging to explain that the state is bi-directionally synced with a file, but then I have a pretty good time with the model using it as a way to "see" and "act," especially if I get myself into a convoluted state while using the app we're working on.

Another spin on this idea is making a separate /playground page with random bits of debug UI, and letting the model see it, and modify as it goes. Usually it takes a bit of extra encouragement to also put in JSON.stringify-ied blobs of data in there, but once a correct "point of view" on the problem is created, the model usually has everything it needs to figure it out.

For example here I was iterating on some tooltip positioning, where it has to take into account its own size, the selection (dashed line), and the size of the viewport:

I'm somewhat confident I could figure out the conditions myself while the model was getting stuck, but it was fun to figure out how just seeing all the possible states at once made it work through the issues. This is interestingly related to Solving Things Visually — seeing the problem correctly can make it way easier to solve, and sometimes the seeing is as simple as throwing in some extra debug UI (which with LLMs is basically "free").

Similarly, figuring out how to "partition" the application into smaller chunks of functionality can be very useful. For example, I sometimes create a separate sub-project where we develop a library with a stable API which the main project then imports, so the model on the main one doesn't get bogged down in the details of some complex but well-known thing.

When building my own Agents for various things, I had very good results by creating a small CLI where each of the sub-tools could be executed by the model that's helping me. This way I can judge the quality of the responses, and iterate on them with the model, without having to go through the whole application loop, which sped up the work on, say, a "summarization" or "data extraction" tool. I let the model run the CLI wrapper, we both look at the responses, I say I'd prefer them to be a bit different, and we continue.

Finally, creating ad-hoc MCP tools for the thing we're working on is always an option. For web-based things this usually makes no sense with MirrorState/Chrome/Playwright MCP already existing, but as I was working on React-Native code, it was useful to let the model click around and take screenshots of the application to test and debug it.

Smaller Explorations

I continued exploring some of the "Codegen Environments" ideas, mainly, adding a way to combine smaller tools into larger wholes.

A box can be drawn around multiple elements, which hides the arrows (treated as "implementation details"), and optionally any extra interface elements created to solve a given problem:

The "boxed" tools can still be connected with each-other using arrows — which can point to either the box as a whole, or a single interface element:

This reminds me of some thoughts from Crosscut on "Encapsulation without lamination". Combining things and exposing only a subset of the internals is incredibly powerful, but seems to always be done in a way where "the whole" becomes the only thing you can interact with.

This also opens up an interesting philosophical question — the act of "boxing" could easily be fractal, so what then is interface and what is implementation? The tools seem to morph from one to the other, or maybe hold a superposition of both, depending on which context they are in...

I also experimented with a pen-based interface, inspired by explorations from Alex Chen:

(check out the Bluesky thread for more videos)

My version, instead of Nano Banana, uses the Flash 3 model with no thinking budget so it responds very quickly, even if often "weirdly". I send the model a composite of my scribbles and the model's previous outputs. As a response, the model generates javascript canvas code, which is added on top of the canvas, and we continue.

The feeling of using this is quite interesting, the combination of human input, machine-styled responses ("Computer Voice"), and quick generation is something I have not seen before in this space. This, of course, smells of Programmable Ink — it's also very fun, but of unclear usefulness at the moment — let me know if you have any ideas where to take this further!

Worth Checking Out

What I've been reading lately:

• The Logic of Chance — was amazing, a style of book I now wish existed for any topic I'm interested in!
• Mind in Motion — more on Embodied Cognition, this time how thought could come up from movement
• Frame Innovation — more Design Science Research and related ideas; basically pitching "re-framing" the problem as the most important part of the design process
• Maintenance: Of Everything, Part One — Stewart Brand and Stripe Press, what more can I say

On the web:

• Computation graphs and graph computation
• code by diagrams, also Paper Desktop app is out with MCP support!
• a semantic project navigator, "just-in-time" code mapping (via Steve Krouse), and Project Copernicus
• Live-Coding (the music kind) in Hazel
• Gyms, Zoos, and Museums: Your documentation should be in-game which seems super related to the Feedback Loops for LLMs above
• talk on "Automat" from Marek Rogalski for the most skeuomorphic VPL you've ever seen
• a new talk from Alan Kay!

the biggest news of this quarter is that I'm an independent consultant now!

I left Replit in October, after 2.5 years. While there, I oscillated between exploring and prototyping, and designing and developing production features. It definitely was a valuable experience to be a part of rapid two-orders-of-magnitude ARR growth. Some of my notable projects are documented on Replit's blog: 1, 2, 3, 4, and the Agent itself.

The company is now focused on creating production applications by prompting, and I've been itching for a while to explore some of the other Open Questions Around LLM Interfaces. Going independent seemed like the best way to do so, and I'm excited to share that I'm currently working with Google Creative Lab, where I'm exploring some of these ideas!

Reach out if you're interested in working together: hi@szymonkaliski.com

In the meantime, I've been making small vignettes around interacting with the language models. I believe that chatting is not going to work for everything, and that there's still a lot to figure out in how to do things with these new fuzzy functions.

Below is a small sampling of these explorations, some of them done on my own time, some at Google.

Codegen Environments

One of the directions I'm excited about is how can a little codegen fit within a Programming System — instead of generating whole applications, can we, by the nature of the environment, get compartmentalization, interoperability, and other interesting properties automatically?

Below is a canvas-based environment where each node is generated by the model from a short piece of text, and the way you use it is a mix of natural canvas interactions: duplicating, resizing, moving around... — with the added communication channel of being able to react to other nodes:

A friend made a good point that instead of asking "to the left of" we could have draggable whiskers coming out of the nodes, and that evolved into a full on operating system of sorts, where every tool is generated on the fly, and I can combine them together by roughly describing what I want:

I also looked a bit into the meta direction of making the process of making visible — which was inspired by earlier experiments with Liunon:

Here, instead of changing the prompt inline, you create modifications on top of the things you already made. I quite like this linear history of progress, but this clearly starts lacking when you want to explore multiple possible directions (which is where something like Spellburst comes in).

Finally, I poked around combining some of these ideas with Programmable Ink, but I didn't get very far:

The main issues were that bounding box detection for detecting similar hand-drawn items remained finicky, and I hit a wall with my vibe-coding skill and would have to get into the code to debug some of the performance issues, which really didn't seem like a fun way to spend a couple of evenings of my time.

Improving Chat

I also played around with expanding the chat interface itself.

Often when I try to learn with the help of an LLM, I want to start follow-up questions from specific passages in the reply:

While this is a complete prototype, I don't really use it — I think partly because I'd have to go to some directory and run npm start, and partly because often I'm already a couple of questions deep in an LLM session, and it's very hard to move that over somewhere else. I'm not sure what to do about this! I could host this tool somewhere, and that would solve the first problem, but the second one is a larger issue — I'd either have to move all my conversations there, or know up-front when something will turn into more of an exploratory session.

If only we had Malleable Software so I could jam this into some existing tool that I already use...

On the topic of learning with LLMs, another idea that I explored was prompting the model to generate explanations together with interactive demos, with an extra twist — these demos have to store their state in a way that's inspectable, and attached as a part of conversation. What that allows for, is exploring some behavior in an interactive way, and then asking follow-up questions about this specific state you're seeing:

I know you could always take a screenshot and paste it back into the chat, but I do wonder why none of the AI providers do this automatically. We can create an artifact, a canvas, or what-have-you, but its run state remains invisible to the model.

Little Tools

A completely different direction that I'm interested in, is exploring small situated "fuzzy" tools:

name-pdfs renames PDF files by their content. I use it a couple of times a week after downloading a bunch of articles.

ai-grep finds things "fuzzily" matching the search term. It nicely complements something like rg (even the output format is the same), and can be useful when I forget the exact wording I used. I use it a couple of times a month, most often when searching through my wiki, or an unfamiliar codebase

ai-translate was an experiment in bi-directional text-to-text translation, it sounded interesting on paper, but I don't use it at all. Still, the idea of translating between code and pseudo-code has stuck with me:

I'm hopeful about the possibilities LLMs open for End-User Programming and creative tools — it's been exciting to revisit the ideas I wrote about seven years ago in this Ink&Switch essay. One of the hard parts is now (at least partly) solved, but a lot of the other ones still remain!

Worth Checking Out

What I've been reading lately:

• Languages of Art
• I went on a little side-quest on education:
  • Theories of Childhood — a very brief high-level overview of the Constructionism/Constructivism thread, I don't think it's really that valuable on its own, but was a nice quick refresher on the general ideas, and how they build up on each-other
  • Mind in Society — highly recommended
  • Changing Minds — also highly recommended, both for the ideas, and for the closer look at Boxer
  • Children Learning by Doing, a paper, not a book, but still fits nicely into this thread, and still highly recommended
  • I also attempted Democracy and Education but I gave up quickly, it was just too hard to read
• I also went through Interface which was a very philosophical and broad look at, well, interfaces, and I've been making very slow progress on Principles of Development — I'm skimming most of it, but I find the ideas fascinating — both the processes happening during the development of the embryo, and how we came to understand them through various (sometimes gross) experiments

On the web:

• an experimental programming language based on topological pattern matching in images
• I love this visual explainer, the arrows pointing directly from the "abstract" to the "concrete" have heavy Crosscut vibes — sadly this is just a static diagram, not an executable one
• Who needs Graphviz when you can build it yourself?
• Game design is simple, actually
• Spatial music-making experience
• some pretty pictures:
  • https://bsky.app/profile/disconcision.com/post/3m55gjuoyl22c
  • https://www.artstation.com/artwork/Nqw4Nb
  • https://x.com/colindmg/status/1978847115779440989
• some videos:
  • Mystery of the Quintic
  • Overhauling a 15-year-old Max patch for Oneohtrix Point Never
  • The Curious Case of Zoomable User Interfaces and AI
  • The Actual Reason Semiconductors Are Different From Conductors and Insulators

Our daughter just turned one!

The days really are long, and the years really are short. She seems to be learning something new every day now, from crawling to standing up, to making sounds, to waving at us — it's been amazing.

MirrorState

MirrorState is a React library and a Vite plugin which handles bi-directional state synchronization through JSON state files. This way the changes made through UI end up on disk in human-readable form, and changes made to these files propagate live to the UI.

MirrorState was almost fully build with LLMs, in, I think, somewhat interesting way:

• I have an AGENT.md/CLAUDE.md file which holds a simple instruction on how to use PROJECT.md
• the PROJECT.md is a mix of high-level "PRD", and a task list subdivided into "features"
• as I was working, I would prompt the LLM to expand what I'm thinking about into a feature, I would then both chat with it some more, and manually edit the file, and once I was done, I'd ask to implement and check off the items
  • I'm honestly not sure if this approach helps with LLM grounding into the state of things, or pollutes its context, but it definitely helps me stay on the right track, and makes sure that we're - at least partially - are "talking" about the same thing - especially when coming back to the project and starting a new chat session

You can try out MirrorState here: szymonkaliski/mirrorstate

What if Iannis Xenakis had Figma?

Iannis Xenakis remains a huge inspiration, I was poking around his graphical notation and made a silly demo to "play" them from Figma:

I learned the hard way that Figma plugins have no support for WebMIDI, which is disappointing. WebSockets do work though, so I just piped the values through a localhost server, but I really don't understand the limitation (I also do understand how some could argue that there's no use for MIDI in Figma).

Pulling Myself Out of the Computer

Over the summer I also managed to discover (yet again) that spending too much time in the computer (or maybe not having anything to balance it with?) is not good for me.

I decided to pick up sketching, and I'm slowly moving through How to Draw, aiming for just a couple of minutes of exercises as often as I can. It's been really fun to spend time with a pencil and a piece of paper, and to move my body in a way that neither barbell nor keyboard forces it to.

I've also been noodling on some music again, which I missed a lot. Optimizing my studio for having a single switch to power on all the gear and be ready to play immediately has really helped with getting just a few minutes in here and there.

Worth Checking Out

What I've been reading lately:

• I skimmed through Algebraic Topology and Topology for Computing — most of it is over my head at the moment, but I do like the pictures, and I have some intuition around what homotopy is, which is nice

On the web:

• Catalogue of semantic zoom patterns
• This is what computers are for
• Fiber optics sculptures
• Code Shaping: Iterative Code Editing with Free-form AI-Interpreted Sketching
• A cool compiler visualizer
• Critical Architecture/Software Theory

Self:

• I published some early unstructured thoughts on Open Questions Around LLM Interfaces, VPLs and LLMs, and "Learning to Program" and LLMs

Edits made through the React state update the file content, and edits to the file update the UI:

MirrorState grew out of a few ad-hoc iterations spread across various side-projects. The main motivation comes from the approach of building prototypes with a single source of truth, which gets hardcoded with initial values.

What often happens further into the project is that I'll take a couple of actions in the UI, and I want to preserve that specific state back into the file — at which point I have to either manually recreate it, or make it available in the console, JSON.stringify, and copy-paste back into the code.

In development, the Vite plugin file-watches all *.mirror.json files for changes, and beams them up through a websocket. The React library connects to that websocket, swaps state on new values from the file, and beams back down the edits made in the UI. In production build, the content of the *.mirror.json file is hardcoded into the build artifact.

Go try it out: szymonkaliski/mirrorstate

While MirrorState grew organically out of my own needs, there are some interesting parallels to:

1. Riffle: Reactive Relational State for Local-First Applications — at some point during its development Geoffrey showed me a demo of opening up the application state in a database editing tool, and being able to inspect and change it as the application runs, which really resonated
2. Code⇄GUI bidirectional editing via LSP — which (obviously) relies on LSP to achieve similar results (livelove is another, more developed, take at this idea)
3. The program is the database is the interface — which is at best an "aspirational" connection, and extends this idea of bi-directionally editable data structure with being able to ... edit itself

In the last issue I mentioned I've been working on some more computer-y side-projects, and here's the first one: Vineyard — an exploration of prototype-based component re-use.

This idea was brewing in the back of my mind for years now — at the moment it still feels undercooked, but I'm curious to get some feedback and hear your thoughts. Go check it out, there's a bit of a longer write-up and a live version to play with.

A friend recently made a dictation app in a weekend.

Here's my 15-minute version:

#!/usr/bin/env bash

set -euo pipefail

export OPENAI_API_KEY="$(cat ~/.openai-key)"

tmp=$(mktemp /tmp/whisper.XXXXXX.wav)
trap 'rm -f "$tmp"' EXIT

echo "Recording... (press Ctrl-C to stop)"
rec --no-show-progress -c 1 -b 16 -r 16000 "$tmp"

[ "$(soxi -r "$tmp")" != "16000" ] && sox "$tmp" -q -r 16000 "${tmp%.wav}_16k.wav" && mv "${tmp%.wav}_16k.wav" "$tmp"

text=$(openai api audio.transcriptions.create -m whisper-1 -f "$tmp" --response-format text)

echo $text | pbcopy
printf "\nTranscript:\n%s\n" "$text"

Adding a global shortcut in Hammerspoon was another 5 minutes on top of that.

Yes, it sends the audio files to OpenAI and requires a network connection — I don't care, for my limited use it's more than ok.

It would be easy to speed up the recorded audio before beaming up to OpenAI to save on some of the cost.

Worth Checking Out

What I've been reading lately:

• The Annotated Flatland (and now I wish every book had a Talmudic rubrication)
• Logic and Design In Art, Science, And Mathematics
• Category Theory for Programmers

On the web:

• Ink&Switch has a new essay on Malleable Software
• Beyond Code Generation: LLM-supported Exploration of the Program Design Space
• What I Learned Building Whole Earth AI
• Design for 3D-Printing
• Semi-Formal Programming
• Scrappy from John Chang and Pontus Granström (with a nice write-up)
• Agentic Engineering in Action with Mitchell Hashimoto

Motivation

Creating reusable components for building applications is a standard industry practice. You don't re-implement a button for every screen of your application, you have a single button component, that's re-used across the code. Maybe all of your buttons are a specific shade of blue, and that's fine, until on one of the pages, you suddenly need a red one, as the on-click handler does something destructive.

A isDangerous property could be introduced, and now all the dangerous buttons are the same shade of red. But what if you also have "warning" buttons, "inline" ones, a "hidden" one just for screen-readers, etc? Maybe it's fine to keep adding properties, or maybe you create a component for each of these uses.

The actual example that motivated this work was when I had a specific well-designed, and well-behaving modal component, used across some project. In one specific place, that modal needed three buttons instead of two, for various reasons. The standard practice would be to decompose the modal into a wrapper that handles some functionality (like displaying as an overlay), some generic body wrapper, and a modal-button section, etc. That's a lot of refactoring to just get a single-use variation to work.

Vineyard is a prototype that explores what it would feel like to be able to override arbitrary properties of any component without extra work.

Project Walkthrough

The leftmost sidebar is a tree of components, organized by ancestry. In this case Blank is the general starting point (a div), then we have a Button component, and Primary is a Button with additional custom styling.

When I make edits to the Button, for example adding an extra border, that propagates to Primary, and all the other children of that component:

I can also make sure that each Button has an icon:

But, decide that I don't want it in the Secondary and Underline styles:

If we come back to the motivating example of building a Modal, here's a general outline of the component:

I can create a variation with three buttons:

Even though I reordered the buttons and modified the subtree of the component, other changes still propagate, like making the default button blue:

Finally, changes I make are recorded in each component, as a side-effect of how the system is implemented:

You can play with Vineyard online.

Open Questions

• It's hard to judge how something like this would feel if used authentically. The system is hermetically sealed right now, and I think a good next step would be some way to maybe export these components into React. Or maybe even somehow just "use" them, without explicit "exporting"?
• It's unclear what should happen when a component is removed. For example, in the demo above we use Button all over the place, what should happen if I just remove that component?
• What about the component functionality? What it would look like to handle button clicks in this system?

References

Vineyard was heavily inspired by Apparatus (a gift that keeps on giving). Even if we ignore the programming model and the solver, the additional compelling idea is how the component system works:

• you can add components to other components, and modifying the parent instance, propagates the changes through the tree, which is fairly standard
• you can also override every property that the component has, without "ejecting" it into the current canvas — and the properties that were not overridden, still continue to propagate (which is also the case in Vineyard)
• even more interestingly, you can do things like reorder the child components, remove some of them, and the property propagation still holds! (which feels fantastic, and is also the case in my prototype)

Self is another system that works this way that I know of, though probably feels less familiar than Apparatus' or Vineyard's property trees.

I'm back to tinkering on some interface-y side-projects, but I need more time to make things legible, and time is hard to find right now.

Surprisingly, I somehow did manage to motorize external blinds in my studio (well, one of them so far) using a car power window motor, a couple of relays, and a bit of 3D printing, enjoy!

Motorizing External Blinds

My basement studio has become a project in itself — there's always something to improve in between other distractions.

One small annoyance is that due to how deep my desks are, and the fact that the windows are behind them, the tape to roll up the external blinds is hard to reach. The room is on the ground level, and I do roll them up and down for extra safety (and the rare occasion when the sun shines directly into my eyes).

I could just buy motorized blinds, but where's the fun in that? Not to mention this being a somewhat larger retrofit, which I didn't want to go through. Instead, I decided to see if I can motorize the current manual system while maximizing the amount of things I repurpose from it. In the end, the plastic housing had to go, but I kept the wall mounts and reused the internal spool, but I'm getting ahead of myself.

The project started by figuring out how heavy the blinds even are. I tied a knot on the tape, attached a portable scale, and pulled until the blinds started rising (~4kg in my case); from this and the radius of the spool I could calculate the torque (~1.2Nm), and from this — find a motor.

Using a wormdrive seemed like a good idea since there's no back-driving — it can stop at any position without supplying any extra power, which is great for this use — the ambition was to be able to not only roll up the blind, but also to keep it up there. Finding a suitable motor wasn't easy (sadly, there's no McMaster in Poland), and the ones I could find with this sort of torque were getting somewhat expensive. I randomly stumbled upon suggestion online that car window power motors are all three: cheap, worm-driven, and pretty powerful. I found a 5USD one (sic!) with enough torque to easily roll up the blind when connected to 12V/6A power.

Electronics were pretty simple as well. Since I wanted to only control the motor direction (or stop it completely) I didn't need any fancy motor drivers, and instead I could use two relays wired like so:

If one is on and the other off, the motor turns; swapping them around makes it turn in the opposite direction; and if both are off, the motor is off too.

I needed some form of feedback on how far the blind has moved, though. I experimented with using Hall sensor and some magnets placed on the spool, but couldn't get reliable readings from it. Part of it is that apparently the ADC on ESP32 is not that great, and part is that I was using a 5V module with 3.3V logic on ESP32, and I was too impatient to play around with a level shifter. In any case an IR sensor with some simple alternating black-white pattern glued on the spool works great — I went with 10 steps per revolution, which is more than enough for this use.

Controlling the blinds happens through MQTT. I have a couple of "debug" routes: one where I can manually move the motor forward/backward by a specified number of steps, and another one where I can set the start and stop endpoint based on the current step (these values are stored in flash so they survive device restarts).

With this, it's pretty straight-forward to create the "end-user" routes: one to get the current position (in 0-100%), and one to set the desired position (also in 0-100%). This is all wired up to a simple Node-RED interface (and HomeKit, through Homebridge), which is what I already use for a bunch of other things. I also connected this to my BLE beacon presence detection. Walking into the room in the morning and the blinds automatically opening never gets old.

Even with everything being simple, this project end-to-end took over two months. I mainly didn't have much time to tinker, but also:

• turns out you need proper wire gauge for 6A (I got to see some exciting smoke once!)
• you also need properly graded connectors (a random barrel jack lying around is not gonna cut it; smoke again)
• ESP32 uses one of the two ADCs for Wi-Fi, which was a fun debugging experience when my sensor was working fine in isolation but not when I wanted to read it through MQTT
• a hard-to-see smudge of glue on the black part of the encoder pattern for the IR sensor was tripping it up, which made me waste two days trying to figure out why sometimes I lose a step or two (solved in two minutes: reprint the pattern and glue it again, just more carefully the second time around)
• the design of the various parts can look good in CAD, but always need a bunch of iterations to get everything to fit perfectly — and even with 3D printing we are far from the quick feedback loop of programming

I didn't take you on this whole journey for nothing though (I know the window is dirty, in my defence the building façade is being renovated):

Misc Stuff

A filament dry box is a fun quick project to make your 3D prints less stringy, using a 5USD container from IKEA, a PTFE tube with a coupler, and 15 minutes of your time. I printed this, drilled a small hole in the container, thrown in a bunch of silica gel and a hygrometer, and closed it up. The hygrometer currently shows a sad face with "too dry" text next to it, so I assume the filament is happy.

As I was debugging the hall sensor, I wished for a nice and fast "digital scope" where I could pipe streaming logs of values and get a zoomable and scrollable chart that I could inspect, like one would do with an oscilloscope. Arduino has a Serial Plotter, but it's a bit oversimplified for what I want — I can't even scroll back to see the history, or set a specific vertical and horizontal scale.

Do you know of any tools that fit this description? Or do I have to Zig and OpenGL on my own?

Worth Checking Out

What I've been reading lately:

• my recent readings on category theory somehow brought me back to computers again — I syntopically went through: Homotopy Type Theory, Types and Programming Languages and Type Theory & Functional Programming; I liked the last one the most, maybe because I red it after the first two ones, so things just looked more familiar

On the web:

• cool prototype combining UI and logic on the same canvas which reminds me of Yoshiki Schmitz's work (who's sadly not with us anymore), like Data Design Language or Dango
• "You don’t program by chatting. You program by writing documents."
• a nice little tool to hack into React internals
• Matt Keeter is at it again (continuously?) with Fidget (and a great talk)
• we shipped Workflows at Replit; I wrote a task runner on top of the previous work of Multiplayer Shell, I'm pretty proud of it — it's used not only by end-users but also by the LLM

This issue covers the last ~six months to catch up for the (understandably) skipped Q3.

It was so great to take time off to fully focus on the family. I'm grateful that everyone is doing well, grateful for autumn that was warm enough for frequent walks, and grateful for not thinking about anything else for a while.

While I was away, we launched two things at Replit that I've been heavily involved in: the Agent — LLM deeply integrated in the platform — and new, orders-of-magnitude faster, and multiplayer-native Shell experience.

I captured some thoughts about why LLMs need IDEs, and how they relate to End-User Programming in this short write-up.

Improving the Shell experience was also a fun project. I built a full vertical slice, from a custom PTY library, up to UI/UX allowing users to run Shell sessions in the background, and collaborate with each other in multiplayer Repls. There's a ton of technical info in the official launch blog post, if you're interested.

Less Eager Copilot

Until recently I've been using LLMs for programming mainly through the vim-ai extension, slightly adapted to my personal needs, sending relevant text back and forth between vim's split views. This meant that I only really used it for bigger "global" tasks — rewriting whole functions, fixing type issues across multiple definitions, etc.

Copilot, as a way for more "surgical" LLM-driven edits, always seemed interesting to try in a "serious context of use" kind of way. What was stopping me, was the "ghost text" UX it relies on, which personally feels over-eager and somewhat distracting.

To work around this, I wrote a small extension for coc.nvim (which I use for LSP integrations) that man-in-the-middle's the Copilot LSP inline completions to, instead, add them to the tab-completion menu:

I've been daily driving this for a few weeks now, and quite like it. Maybe it's useful to some of you too: coc-pilot.

Organizing with 3D-Printing

I use an IKEA BROR utility cart for a workbench in my basement studio. It's pretty small and quickly gets messy when I'm doing something, making it hard to access my most-used tools.

A pegboard is an obvious answer for nerdy organization, and the one from IKEA almost fits their BROR system — if only it was a couple of centimeters wider, I could simply connect them directly.

Instead, I had to adapt an adapter I found online to do so.
My variation is here, again, maybe useful to some of you.

I wish IKEA was more like Lego. There must be a reason why their furniture (especially the "utility" lines) is not a bit more standardized. I don't think it's "vendor lock-in," since the standardization I wish for is just across the IKEA stuff, which would lock us in even more. Let me know if you have any ideas?

On the 3D-printing topic, the pegboard attachments from IKEA are not that versatile, but there are (of course) a ton of people making custom ones.

Worth Checking Out

What I've been reading lately:

• Simulation and Its Discontents
• and more math:
  • Mathematics and Its History
  • Basic Category Theory for Computer Scientists
  • Category Theory In Context
  • Topology

On the web:

• A collection of talks on UIs and LLMs
• I love how the documentation here is executable, and part of the tool
• "A Genealogy of Technology and Power Since 1500"
• "I built myself an ISO 5 certified positive pressure cleanroom in my basement to pursue independent research"
• Live values in your editor
• Probabilistic spreadsheet UI
• Programming-by-example with LLM
• "If you need the money, don't take the job"
• "Hobby CAD, CNC machining, and resin casting"
• How to shine LEDs through PCB
• "a toy computing system based on (...) Dynamicland’s Realtalk"
• Information Display Systems for Soyuz Spaceships
• more UIs should incorporate "weird little guys" *
• regex visualizer

Fin

Best wishes for 2025!

I'm taking a break for a couple of months.

Replit is an online software development platform, accessible through the browser, and backed by virtual machines running in the cloud. This cloud-nativeness provides unique advantages over browser-based IDEs which run the code in the browser itself. Almost coincidentally, these advantages can be also useful to LLMs, which created a foundation for developing an autonomous Agent, capable of creating software on the platform.

End-User Programming

The target user for the Replit Agent is not a professional software engineer, but a "local developer" (after Bonnie Nardi), or a "barefoot" one (after Maggie Appleton). This, in many ways, makes this work a part of the End-User Programming canon.

For a long time, my personal hope was that we're going to create/discover a magical programming system (a system, not a language) with clear heuristics, and an interface that makes programming not feel like programming.

For a while after LLMs started getting better at outputting code, I thought that none of that matters anymore — if the code can be generated based on requests stated in plain English, why would we care about anything other than the results of executing it?

After working on the Agent, I believe that the original learnings are as important as ever. We still need a programming system, one where humans and LLMs can collaborate on creating software. The interfaces matter even more so, as there's one more "player" in the mix now — in addition to the user interacting with the environment, there's the LLM which interacts with both.

IDEs for LLMs

LLMs are pretty good at generating (specific kinds of) code, and only getting better. But, maybe unsurprisingly, just throwing code at the problem doesn't lead to the best results — the LLM operates blindly, with no feedback from the environment. Turns out, LLMs can be more powerful, if given proper tools.

A lot of work on this project was centered around giving the Agent well-designed interfaces to the environment (and to the user), for example:

• exposing a custom "install packages" tool gave us more control to handle various edge cases, when compared to allowing the LLM to run random packager commands directly in the shell
• exposing a "run a command in the background" allowed us to have control about the execution loops of both the Agent and the command in question, so we can keep them in sync, and surface relevant outputs to the LLM
• giving the Agent an "ambient awareness" of issues found by LSP, outputs from the running commands, screenshots of the webview outputs, etc. allowing it to course-correct as a part of its work, often without needing any additional human assistance
• allowing the Agent to "present the results of its work" led to better interactions than trying to rely on heuristics (like opening a webview when detecting a newly opened port, which had a lot of false positives)

Interestingly, we didn't build almost any of these tools from scratch — our IDE already had them. We only had to surface and package them for the Agent. In many ways this is an UI problem, just that the user in question is an LLM, and the interface has to be synchronous and textual.

In this project, I was responsible for early research, design, and prototyping, both for the user-facing interface to the Agent, and for the Agent-facing interfaces to the environment and to the user.

The prototypes had a lot of influence over the final project, where I also contributed production code, ranging from implementing some of Agent's tools to building parts of the user interface. *

This is the fourth technology that I've used here, and I finally decided to take the whole thing into my own hands. The previous version of this site was made on top of Gatsby, and I don't really remember my reasoning for picking it, other than wanting to migrate off of Tumblr, where I migrated to from Wordpress, over a decade ago. There's nothing wrong with any of those, it's just that I kept on making custom plugins and hacking around what Gatsby exposes, and it got to a point where the whole thing felt both very brittle, and very complicated. On a somewhat philosophical level, I also don't think I need any frontend JavaScript for some text and an occasional image.

High-Level

The way the generator works is the most straight-forward approach I could come up with. There's an input directory, for example:

├── assets/
│   └── photo.png
├── writings/
│   └── 2024-07-01-static-site-generator/
│       ├── index.md
│       └── miniature.png
├── feed.xml.ts
├── style.css
└── index.tsx

I recursively traverse it, process every file I find, and output the same structure:

├── assets/
│   └── photo.png
├── writings/
│   └── 2024-07-01-static-site-generator/
│       ├── index.html
│       └── miniature.png
├── feed.xml
├── style.css
└── index.html

Most of the behavior can be inferred from the difference between the two:

• index.tsx becomes index.html using React's renderToStaticMarkup
• index.md also becomes index.html by simple process of parsing markdown into mdast, into hast, into React tree, and renderToStaticMarkup-ing as well
• feed.xml.ts becomes feed.xml; it's a .ts file, and in this case I just call the export default from there, and write its result to the output file
• images are lightly optimized, with their preview base64 representations injected into the relevant html files
• everything else (like style.css) is just copied over

This way my deploy script is just an rsync to a server.

The "fun", as always, is in the details:

• Normalizing Markdown
• Backlinking
• Transclusions
• Time-Tracking

Normalizing Markdown

Most of my content (projects, articles, newsletter) is written in markdown. Notes are also in markdown, but with a slightly different syntax, as they are generated from a subset of my personal wiki. For example, in the website-first markdown I keep a YAML header with metadata, when I link between things, I use the url's instead of relative file paths, etc.

Instead of building a generic system to handle these differences, I just have a different code-path for when the markdown file is inside the notes/ subdirectory or not. This is the biggest upside of a totally custom, one-off solution: I don't have to change how I write my personal notes, or figure out how to plug into someone else's plugin system; I just write some custom code when I need it.

Backlinking

If you clicked around my website, you probably noticed the backlinks section at the bottom of most pages, for example here. To generate this section, when I parse the markdown, I find all of the local links, and store them together with other metadata about that page. I also grab the AST subtree containing a "meaningful" nearest paragraph containing the link.

When rendering the final pages, which happens after the input directory has been fully scanned, I can find items which mention the page that's currently being processed, by looking for a match between what that other page links to, and what is the current slug.

Because I also keep the "meaningful" paragraph, I can generate the Text Fragments built into modern browsers, and highlight the backlinked sections! To try it out, go here, and click on, for example, the "Protoboard" backlink — you'll end up in the middle of that article, with a relevant paragraph highlighted.

Transclusions

I often publish smaller demos as a part of my newsletter, and they don't end up on the projects page, which feels like where you'd look for, well, projects.

I could just copy-paste the relevant text between these two places, but that would be too simple — instead I've implemented a way to transclude markdown sections using custom markdown directives.

To do this, I use remark-directive to parse custom :::transclusion{slug, header} directive: slug points to the page from which to transclude from, and header to the header of the section to copy over.

When I find that directive in the markdown that I'm parsing, I have to transplant a subtree of the other page's markdown AST, so I have to already have it in memory. Hence the website generation is done in three passes:

1. Initial scan of the input directory, and reading all the items into memory, including initial markdown parsing, but not replacing the transclusions yet.
2. Process the transclusions, and grab all the links from markdown files for backlinking. I do this after transcluding, so the links from that material are also included.
3. Iterate over all the items again, this time generating the final output files.

For a demo of transclusions, check out the Liunon project.

Time-Tracking

The previous version of this website had a separate page with some public-facing stats, just for fun:

This page was using the data from my time-tracking system, and was implemented in a pretty hacky way — I had a small API server which would collect the Google Calendar events, and post-process them for rendering. The client code would query that server and render the charts — the JSON blob was huge, and the client had to do the work every time the page was opened.

I decided to make the stats a bit less real-time, instead of updating every 10 minutes or so, I'm updating them when the static site generator runs. With that, I also reworked this section a bit, which you can now see on the main page.

It feels really nice to not have to run any client-side JavaScript to generate this. Since the code runs "on the server" (on my machine when the website is generated), I can simply import parts of my time-tracking library, and call whatever functions I want, which ultimately synchronously accesses my filesystem, parses some JSON's, and calculates values for the charts, which get embedded as static <svg>'s inside the html file.

Conclusions

In most cases it's probably better to grab something off-the-shelf, or even just write HTML manually, and move on with your life. I'm not sure if this was the uncommon case, but at least I can check another one of the nerdy "achievements" off my list.

The project took more time than I would have liked, mainly because I had to deal with a lot of idiosyncrasies around migrating from the previous system, and at the same time, I was fixing various styling issues that I accumulated over the years, and adding a couple of features. Did I have to do it all at once? Probably not, but here we are.

Overall, I'm quite happy with this setup, and I also hope I won't have to do this again.

The only thing I can report on this quarter is that I went down one of the most common computer-person-with-a-website rabbit holes imaginable: I wrote my own static site generator. At least, I didn't do it to procrastinate actually writing something, but to keep this site in a healthy state.

This website is now just plain HTML & CSS, just how Tim intended. It is generated from ~2000 lines of bespoke Typescript code. I documented the details in a little write-up — there are some fun things in there: custom markdown directives, backlinking with highlighted text fragments, and cross-markdown transclusions. Enjoy!

Worth Checking Out

What I've been reading lately:

• I'm continuing to progress slowly on the mathematics thread. I finished Concepts of Modern Mathematics which I really enjoyed, and I went through All the Math You Missed (well, mostly just skimmed); I think I want to learn a bit more about category theory, so I ordered the classic Conceptual Mathematics.
• I also read Uncontrolled, and I mostly enjoyed the first, more philosophical part.
• I've been also making my way through The Innovator's Dilemma, The Origin of Wealth, and How to Be Idle (How to do Nothing resonated more with me though).

On the web:

• Apple held another WWDC and demoed Math Notes, which seem at least a bit related to the Programmable Ink research at Ink&Switch, though I think the similarity is fairly superficial — both do have some form of programming and drawing on the same canvas, but in Apple's universe that's all that it is: changing the numbers doesn't impact the drawing, or the other way around. They are as disconnected as having a code editor on one side of the screen, and a scanned drawing on the other. I'm also not a fan of a machine trying to imitate your own handwriting, I think it should have its own voice.
  • On that topic, Lu has been exploring a lot of related ideas at tldraw: 1, 2, 3, etc. All worth having a look (and following!) if you haven't see it yet.
• I'm not sure what this is, but it sure has nice UI
• You don't know this yet, but you do need this 1h23m YouTube video of someone overengineering a DIY oscillating belt sander in your life
• Diffusion On Syntax Trees For Program Synthesis
• Tyler is iterating on Spellburst
• CADmium: A Local-First CAD Program Built for the Browser
• Toolpaths projected on top of CNCs
• "Lode Runner" was written in a ... pdf

Fin

Summer has finally begun, which means that once again, I'm optimizing for swimming in the nearby outdoor pools as often as possible. Life is also keeping me busy, so I make no promises about any new side-projects for the next issue, but I do have a couple of things in the back of my mind.

In the meantime, if you find any issues with the new website, do let me know!

A lot has happened this quarter; actually, seems like a lot is happening every quarter. Maybe three months is just a long time?

This time I write about moving my studio to the basement, our "smart" home setup, and give you a short experience report from setting up Nix on macOS.

Moving To The Basement

Ever since properly joining the workforce, I've been always working from home, which I acknowledge is a privilege, but not without its downsides. I've heard multiple people struggling with "the couch is right there", but I had the opposite problem: "the work is right there", and there's always something more to do. It was always hard to finish my workday, and the timezone shift to the west coast doesn't really help either (all my coworkers are only just starting their day). Also, home can be distracting sometimes.

For a long while, I dreamed of having my own separate space, one that I can lock up and leave at the end of the day, one that's quiet when I need it, one where I can be noisy when I want to, and one where I can be messy — where I don't have to worry about dropping a screwdriver on the floor, or having to immediately clean up when I drill into an aluminum extrusion.

At the beginning of March, this dream came through. The building we live in had a small doctor's office, which has been unoccupied for a good couple of years now. We managed to rent it, and spent the last year renovating: first demolition and clearing up everything we could, and then light polishing pass — new floor, new bathroom — and finally two weekends of running up and down the stairs to move from here:

To here:

It's only been a month since I've moved, so definitely not enough time for any sort of long-term reflection, but I definitely feel a clearer separation between when I'm "at work" and when I'm not, which is great.

Smart Home Setup

Moving to the basement is a good segue into talking a bit about the minimally-smart home setup we have, as the move allowed me to add some fun automations.

My general philosophy here is that the IoT stuff should have no access to the outside world (internet blocked on the router), and that the automations, if any, should be as straight-forward as possible.

The setup consists of:

• Raspberry Pi (not even one of the new ones) with a Zigbee adapter for one IKEA led strip that we use for ambient light behind our TV.
• DIY mmWave presence sensor wired to Raspberry Pi GPIO, also for our living room. These sensors are better than PIR (though more expensive) since they can reliably tell you if someone is there or not, instead of just letting you know when there's movement detected.
• A bunch of Wi-Fi sockets flashed with Tasmota, and with blocked internet access.
• A couple of Reolink cameras, dynamically blocked from the internet (more on that below).
• DIY BLE beacon detector for the studio made with ESP32 board.
• lgtv2mqtt2 for control of our TV (which also has blocked access to the internet, but is connected to a local network).

Everything is wired with Node-RED, and the state is kept in MQTT, so even if I restart the Node-RED process, the current state stays alive, and I can monitor and access it in multiple ways. We also have a very basic Node-RED Dashboard so we can toggle stuff manually when we're at home (and I can do so while I'm away too, thanks to the magic of Tailscale; I wasn't so cruel as to force my wife to deal with this though).

The only automations we have set up at home are turning the LED on at dusk if anyone is present in the living room, and turning it off at specified time at night.

The studio automations are bit more fun, since I'm the only one having to deal with them if they misbehave:

• I have an ESP32 monitoring for a presence of a specific BLE beacon attached to my keychain, so the system knows if I'm in the studio or not. Based on that, the camera and a dehumidifier turn on or off (in anti-phase to my presence, the dehumidifier is noisy). The ESP32 code is very simple: every 30 seconds it starts a scan looking for that beacon, and once it makes double-sure it's there or not (there are some false negatives), it updates the MQTT state over Wi-Fi, and Node-RED automations take over. Let me know if you have any other ideas for reliable presence tracking. I was thinking about using RFID, but seems too finicky to have to place my keys exactly in the right spot every time, and even more finicky to figure out how to detect them in a long-range (which is possible with UHF RFID, and they can even be positioned in space with additional trickery).
• When I turn on my speakers, my audio interface also turns on — but not the other way around (I might want to work in headphones). This is mainly because the speaker switches are not easy to reach.
• I also have a script on my laptop that starts my music production / music listening software of choice, and makes sure the speakers and audio interface are on.

Going back to our home, the cameras we have in our living room have access to the internet only if we flip a switch in the Node-RED dashboard. We basically "turn them on" only when we know we'll leave our dog inside for a while, or if we're going on holidays. It would be great to automate this, but I can't come up with a way that wouldn't be annoying to my wife, so here we are, flipping virtual switches like cavemen.

This is all done just for additional peace-of-mind, we have a proper alarm that has nothing to do with this system (so I don't break it when I'm messing around with things).

The internet access is blocked on a router, by SSH-ing into it from Node-RED, executing some nvram commands, and restarting the firewall service, it's terrible, but it works (SSH is enabled, of course, for LAN access only). For those of you who are very curious, we have ASUS RT-AX86U (actually two of them, one in our apartment, and one in the studio, hard-wired with an ethernet cable), and the relevant commands are:

nvram set MULTIFILTER_MAC="MAC_1>MAC_2>..." # yes, ">" is the separator
nvram set MULTIFILTER_ENABLE="2>2>..."      # yes, "2" means "blocked"
nvram commit
rc rc_service restart_firewall

I got these keys by running nvram show, changing some options in the UI, running it again, and diffing the output.

The important part is that everything still works even if our internet connection is not working, and even if the local network drops, the only thing we lose is being able to control the IKEA led, everything else is done through Wi-Fi sockets, which have physical buttons on them (though sometimes hard to reach).

Setting up Nix on macOS

Continuing with the nerdiness, working at Replit was a forcing function for taking a closer look at Nix (since the Repls, and our repos, both use it for package management), which, turns out, is actually pretty nice.

I got double-encouraged by Homebrew breaking some Python stuff (again), and one weekend I sat down and migrated to Nix on macOS which was surprisingly painless:

• I started by installing Nix using The Determinate Nix Installer, which has some nice properties over the stock one (like being able to easily uninstall it if I want to).
• Then, I roughly followed this tutorial, stopping at home-manager managing my dotfiles — I might get around to adding this at some point, but I just didn't want to change too much at once.

The "developer experience" of direnv combined with Nix flakes is great: you just cd into the project, and all of the necessary dependencies are not only auto-installed, but also persisted in a fully reproducible way (well, provided you wrote the flake.nix for them, which really doesn't have to be that complicated).

I can now have different pinned Python versions for different projects, which solves my original issue. I also got rid of nvm since I can get specific Node versions from Nix too, and it just feels nice to have all of that taken care of by a single system.

If you're curious about my pretty straight-forward configuration, you can browse through my dotfiles repository.

Worth Checking Out

What I've been reading lately:

• I've been slowly continuing my mathematics thread, first going through half of Mathematics for the Nonmathematician which just wasn't picking up enough speed for what I was looking for, so I switched to Concepts of Modern Mathematics, which has been great so far!
• I also made it a personal mission to finish all the remaining books I had with the phrase "The Mind" in the title: Frames of Mind (a classic on "multiple intelligences"), Models of the Mind (which was a bit too "pop-science-y" for my taste, but could be a good resource to pull some threads from), and Infinity and The Mind, which, as the title suggests, takes a look at the concept of infinity from a ton of different points of view.

On the web:

• School Is Not Enough
• Ink&Switch has been publishing regular, shorter write-ups on their work on Patchwork. For more "productized" approaches to improving version control experience, check out GitButler, and a nice write-up of how one person uses git worktrees. For more "huh that's interesting", check out how Ableton diffs Max/MSP patches.
• More cool concepts for improving the shell experience. On that topic I shipped inline AI generation for shell commands, and have more shell-related things coming relatively soon, like this two-orders-of-magnitude performance improvement.
• Francois has been working on bringing tldraw to VS Code, which looks pretty cool.
• In-depth guide to CNC Lasers.
• Another cute DIY MIDI controller, this one with endless knobs and LED rings.
• There's recently been a bunch of chatting around what a "design engineer" is/does on Twitter (1, 2, etc.), and I somehow even ended up on a list made by Maggie, which felt nice. Thank you, Maggie! I have some thoughts about this, but not that many — on some level, I feel like all the "is X(hamburger / Pluto / ...) a Y(sandwich / planet / ...)?" sorts of questions are exercises in trying to find a single all-encompassing ontology to fit everything, and I'm not alone in believing that there isn't one. Additionally, the only time when defining what I do mattered, was when I was looking for a new job last year, and even then, it was more of a negotiation with whomever was on the other side, trying to build a definition that was useful in that specific context. At Replit (and before that too), I just do what I believe is important, and it doesn't matter much to me if that means writing a proposal, recording a video, designing something in Figma, or implementing it on the "frontend" or "backend" (oftentimes it's all of these).

Fin

As always, let me know if you have any thoughts about anything. You can reply to this email directly, or reach out to hi@szymonkaliski.com

And in the meantime, have a great spring, and see you in three months!

Tinyletter had an "incredible journey" moment recently, so if you're subscribed, you're receiving this through Buttondown — more expensive than the free Tinyletter was, which hopefully is a good thing for not having to migrate any time soon again. If you relied on the archive, you can always read the past editions on my website.

I spent the last quarter working on improving Replit's Console, and noodling on a toy database-y creative coding environment, more below!

Improving Replit's Console

I redesigned and rebuilt Replit's Console (almost) from scratch. A single terminal view was replaced with a virtually scrolling list of terminals, where each execution happens in its own context, is reliably cleaned up, even if the process misbehaves (thanks to the magic of cgroups), and which is already starting to integrate more elements of the IDE into this single pane.

Check out this blog post for the reasoning behind the project, and various implementation details.

Throughout this work, two things were often on my mind:

• One, how the line between frontend and backend doesn't really make sense, and how the only line that matters is between the user and the machine. A lot of "UI" and "UX" in this project stem from deep backend work, and wouldn't be possible to achieve by only redesigning the frontend bits.
• Two, how so many details are important to get right (and how often a detail actually ends up being a big engineering effort) for the tool to "disappear in your hand". For example, the behavior of blocks growing with their content up to the point where they take over the whole pane, and the gesture of scrolling up "just works" where in reality, the logic is pretty complex — which reminds me a bit of the infamous "how Slack decides to send you a notification" diagram.

Database-y Creative Coding Environment

Outside of work, I've been noodling on a new creative coding environment, in which you generate images by querying the existing state, and asserting a new one, in a sort of transactional way, with the hope that each of these transactions will also have a meaningful representation, so seeing the intermediate steps requires no additional work, it is the work.

Let's see how we could create something resembling the Space Colonization algorithm in the tool.

First, we assert that there's a page to draw on, place a bunch of attractors, and a single node in the middle:

Second, we give each of the nodes two "zones" — one which "growth" is directed by, and another, smaller one, where the attractors are "consumed". So far so good:

Third, we have to grow towards the attractors by creating new nodes, and consume them if they are in the consumption zones:

This big step does a lot, all at once, which doesn't really help much with understanding what's going on. There's iteration, there's a lot of state, and a lot of playing computer in your head.

To make it worse, we also loop this step 20 times, to let the structure grow out a bit (with a simple number picker out of the frame), and you can only see the last step of that process.

Finally, we can drop all the circles, leaving just the lines...

...which we can export, convert to G-code and plot on the DIY Pen Plotter, which was my plan all along:

Let's go back to the middle step — as you probably realize, it is the one that I'm pretty unhappy with. It feels as if I've been struggling with some version of this problem for a while now — representing and visualizing single values is not that hard, but as soon as collections come into play, the simple solutions fall apart.

We hit this earlier on in Inkbase where interacting with a single item was nicely visualized with inspector panes, but groups of objects were not. We tried to explicitly solve this in Crosscut, but the solutions were far from ideal. I also hit this in my research at Glide where displaying lists of values is the main thing you do.

So far, I've seen two interesting approaches to solving this:

• The spread operator in Apparatus, which is well illustrated by the Bar Chart example — notice how the X property spreads between 0 and 10. It is a single property, but represents a collection, and each of the other operations "just work" (you can sin(X) and get back a spread of values — I've heard that this is pretty much a "List Monad").
• Engraft by Josh (and earlier Pane) both have map behavior as part of the core interaction, with each step being visible and represented on the screen. Engraft is especially promising — an attempt at building an environment where various live, visual, domain-specific tools can coexist and communicate with each other.

One more random thought — the idea of "rubbing a database" on a creative coding library could be an interesting tangent to explore on its own. For example, imagine having computed facts that could be queried too, like intersections, nearness, etc. — which feels similar to how Realtalk works (a database-OS powering Dynamicland).

Anders Hoff a.k.a. Inconvergent has also been exploring home-grown Datalog for making computer art, which is worth checking out (both for the computing, and the art).

Worth Checking Out

What I've been reading lately:

• I circled back to some readings on Embodied Cognition (all great!):
  • Being There
  • Supersizing the Mind
  • Cognition in the Wild
• I started a thread on properly (re-)learning mathematics with A Programmer's Introduction to Mathematics — I felt for a long time that I might not have paid enough attention in school to some important connections, and while I do know some things here and there, I don't have a proper "structure" on which to hang them; I accept that this will be a longer learning project, and so far it actually feels quite nice to be going through proofs and exercises.

On the web:

• A cool live multimedia spreadsheet for programming
• Origami crease patterns
• Annotated lambda diagrams
• Elliot is at it again
• The Gulf Between Design And Engineering
• a bunch of great UI ideas in this new DAW — especially around spatial queries, and the mechanics that allow one to derive a delay effect from interactions between send and return blocks
• a really nice write-up on Spellburst and UIST 2023

Fin

Let me know if you have any thoughts about making loops Live, Rich, and Composable, or about the intersection of databases and creative coding.

Best wishes for 2024!

Transcluded from Database-y Creative Coding Environment

Outside of work, I've been noodling on a new creative coding environment, in which you generate images by querying the existing state, and asserting a new one, in a sort of transactional way, with the hope that each of these transactions will also have a meaningful representation, so seeing the intermediate steps requires no additional work, it is the work.

Let's see how we could create something resembling the Space Colonization algorithm in the tool.

First, we assert that there's a page to draw on, place a bunch of attractors, and a single node in the middle:

Second, we give each of the nodes two "zones" — one which "growth" is directed by, and another, smaller one, where the attractors are "consumed". So far so good:

Third, we have to grow towards the attractors by creating new nodes, and consume them if they are in the consumption zones:

This big step does a lot, all at once, which doesn't really help much with understanding what's going on. There's iteration, there's a lot of state, and a lot of playing computer in your head.

To make it worse, we also loop this step 20 times, to let the structure grow out a bit (with a simple number picker out of the frame), and you can only see the last step of that process.

Finally, we can drop all the circles, leaving just the lines...

...which we can export, convert to G-code and plot on the DIY Pen Plotter, which was my plan all along:

Let's go back to the middle step — as you probably realize, it is the one that I'm pretty unhappy with. It feels as if I've been struggling with some version of this problem for a while now — representing and visualizing single values is not that hard, but as soon as collections come into play, the simple solutions fall apart.

We hit this earlier on in Inkbase where interacting with a single item was nicely visualized with inspector panes, but groups of objects were not. We tried to explicitly solve this in Crosscut, but the solutions were far from ideal. I also hit this in my research at Glide where displaying lists of values is the main thing you do.

So far, I've seen two interesting approaches to solving this:

• The spread operator in Apparatus, which is well illustrated by the Bar Chart example — notice how the X property spreads between 0 and 10. It is a single property, but represents a collection, and each of the other operations "just work" (you can sin(X) and get back a spread of values — I've heard that this is pretty much a "List Monad").
• Engraft by Josh (and earlier Pane) both have map behavior as part of the core interaction, with each step being visible and represented on the screen. Engraft is especially promising — an attempt at building an environment where various live, visual, domain-specific tools can coexist and communicate with each other.

One more random thought — the idea of "rubbing a database" on a creative coding library could be an interesting tangent to explore on its own. For example, imagine having computed facts that could be queried too, like intersections, nearness, etc. — which feels similar to how Realtalk works (a database-OS powering Dynamicland).

Anders Hoff a.k.a. Inconvergent has also been exploring home-grown Datalog for making computer art, which is worth checking out (both for the computing, and the art).

My hope is that this collection of notes will be useful to you if you ever embark on a similar project:

• Designing
• Materials
• Electronics
• GRBL
• G-Code

Designing

Up to this point, all my designs for 3D-printed objects were either made in code (using my own tools, of course: Modeler and hiccup-sdf) or using SolveSpace.

The code-based approach is great for parametric designs or art projects (like FabFungus), while SolveSpace is awesome for quick sketches. In this project I had to figure out how to combine multiple parts together, and there was just no way around trying a serious tool — I went with Fusion 360 just because I had heard about it previously.

You can easily find a ton of "get started with" tutorials for Fusion 360 on YouTube — I can't point to one that was the most helpful. I just watched a couple of them and started playing around with the tool (which is free for personal use).

When designing 3D-printed parts to connect real-life objects, you'll want to reference these objects in CAD. You can model them yourself, but usually it's possible to find them in the GrabCAD Library. For Fusion 360 you want .stp files, which you can add to your project. Here's a tutorial from Autodesk.

3D-printed objects tend to shrink after they are printed. Nothing in reality fits as perfectly as what you design in CAD. If you make an exact 20mm by 20mm hole for an aluminum profile, there's a big chance that it just won't fit after the print. What I did, was print a couple of draft versions of the part with various sizing adjustments to test what fits exactly right, and then used that throughout the design. For example, in my setup, this meant printing 3.2mm instead of 3mm holes for M3 screws, 8.4mm instead of 8mm for the rotating trapezoidal screw, etc.

For Fusion 360 itself, I recommend adjusting two settings:

• Turning on native trackpad support for a better trackpad experience.
• Disabling graphic effects for much better performance.

As for the design itself, I wouldn't recommend going for wheels for the Y-axis, as you can see in the picture above. It's very hard to get the tolerances just right, and my Z-axis is very wobbly. This is fine for a pen-plotter, but a no-go for CNC. I'm planning on redesigning this to use two linear rods with linear bearings, which should create a much more rigid structure than the one I have for the Z-axis.

Materials

The way most DIY 3D-printers, pen-plotters, and CNC's are built is by combining widely available prefabricates with 3D-printed objects.

I designed the body of my pen-plotter using:

• 20x20mm aluminum extrusions (often called just 2020)
• 10mm linear rods and matching 10mm linear bearings (LM10UU)
• 8mm ACME lead screws with matching lead screw nuts
• NEMA17-clones for motors
• aluminum couplings for connecting the motors with the lead screws

Another option for transforming the motor's rotation into linear motion is by using belts, but I decided that the lead screws would be better for the CNC, as intuitively, it feels like more force can be transferred this way. Also, I don't have to worry about tightening the belts.

For getting these parts, 3D-printer shops are your best bet. Motors can get quite expensive, and I just went with the cheapest ones I could find. The aluminum extrusions, rods and lead screws you can get cut to size (or cut on your own if you have the proper tools), but I managed to just find ones that fit my build (and sometimes I designed around what I could find).

For connecting the extrusions to 3D-printed parts, I used the standard approach of adding T-nuts in the profiles, leaving holes in the design where the T-nut should be, and screwing them together. If you've ever assembled a 3D-printer, you know what I'm talking about, if not, just search YouTube for T-nuts, I guess.

The linear bearings are sandwiched between two 3D-printed parts that I screw together. This was the part of the design that I had to do the most trial-and-error to fit perfectly.

Electronics

To move the motors, you'll need a couple of electronic parts. I went with what seems to be a standard setup:

• Arduino CNC Shield
• four A4988 motor drivers
• cheap Arduino UNO clone I had in my drawer
• a power supply — I had a 3A 12V one in my drawer too, and it seems to do just fine
• end-stops — anything that you'll find in your electronic store of choice should be just fine, I went with the cheapest mechanical ones I could find (two per axis, so six in total)

You'll want to set up VRef for your motor drivers — again, there are a ton of tutorials about that on YouTube. This is mainly so the motors don't overheat.

End-stops are a safety feature, so the machine gets a "hard stop" when it physically hits a certain limit (instead of destroying itself). They can be "normally opened" (NO) or "normally closed" (NC). I went with "normally closed", as they are safer — if a wire connection breaks for whatever reason, the result will be the same as if the end-stop was triggered. With "normally opened" ones, we don't get that safety.

Properly wiring two "normally closed" end-stops per axis wasn't immediately obvious to me, especially going through the CNC Shield. What seems to work is wiring them in sequence, connecting one end to the axis end-stop pin and the other to the ground, like so:

To debug the motors and end-switches wiring with the CNC Shield, it's best to use simple ad-hoc Arduino code instead of going for a full GRBL install, as it introduces another layer of indirection and makes it harder to pinpoint where the issue might be coming from.

Below is the code I used to see if I could move one motor. To test different motors, adjust the dirPin and stepPin which you can find by looking for the CNC Shield pinout.

#define dirPin 5
#define stepPin 2
#define enablePin 8

// adjust these based on the motor spec
#define stepsPerRevolution 200
#define microsPulseWidth 250
#define microsBetweenSteps 1000

void setup() {
  pinMode(stepPin, OUTPUT);
  pinMode(dirPin, OUTPUT);
  pinMode(enablePin, OUTPUT);
  digitalWrite(enablePin, LOW);
}

void stepMotor() {
  digitalWrite(stepPin, HIGH);
  delayMicroseconds(microsPulseWidth);
  digitalWrite(stepPin, LOW);
  delayMicroseconds(microsBetweenSteps);
}

void loop() {
  // rotate clockwise
  digitalWrite(dirPin, HIGH);
  for (int i = 0; i < 5 * stepsPerRevolution; i++) {
    stepMotor();
  }

  delay(1000);

  // rotate counterclockwise
  digitalWrite(dirPin, LOW);
  for (int i = 0; i < 5 * stepsPerRevolution; i++) {
    stepMotor();
  }

  delay(1000);
}

And here's what I used to debug the end-stops:

#define xLimitPin 9
#define yLimitPin 10
#define zLimitPin 11

void setup() {
  Serial.begin(115200);
  pinMode(xLimitPin, INPUT_PULLUP);
  pinMode(yLimitPin, INPUT_PULLUP);
  pinMode(zLimitPin, INPUT_PULLUP);
}

void loop() {
  Serial.print("x = ");
  Serial.print(digitalRead(xLimitPin));
  Serial.print("y = ");
  Serial.print(digitalRead(yLimitPin));
  Serial.print("z = ");
  Serial.println(digitalRead(zLimitPin));

  delay(200);
}

GRBL

Once all the axes move and the end-stops are wired properly, it's time to start controlling the machine with G-code, the standard for 3D-printers and CNC machines.

Luckily, there's an open-source G-code interpreter for Arduino UNO and the CNC Shield called grbl. The installation is pretty straight-forward, and described here — you basically add grbl as an Arduino library and then upload the only sketch that it provides: GrblUpload.

There's one adjustment I had to make in config.h, the configuration for the controller — I commented out #define VARIABLE_SPINDLE which conflicts with Z-axis end-stops.

Once grbl is running on the Arduino, the real fun starts — configuring all of its settings. You can do this directly through the Arduino serial console, but I decided to switch to a GUI, so it's easier to test and control the machine. I went with cncjs, after testing a couple of other apps that didn't seem to work correctly for whatever reason.

For the configuration, there's a lengthy documentation on the grbl wiki; here are the things that I did:

• calculated steps per mm for the motor movement, as described here
• adjusted maximum rate for the motor's, as described here
• inverted some of the motors axes to move in a predictable way
• enabled end-stops:
  • they had to be inverted since they are "normally closed"
  • they had to be enabled
  • and with this, the homing cycle (automatic corner finding) can be enabled too
• finally, I homed the machine (using the button in the GUI), and stepped it manually to the other end-stop to figure out the soft limits (which also have to be enabled)

G-code

To plot anything, the machine needs G-code describing the sequence of motor moves. There are many ways of generating it, the one I went with was vpype and vpype-gcode — a powerful command-line tool for everything SVG and plotters.

Since this is a completely custom pen-plotter, I had to create my own configuration for vpype-gcode, which looks like this:

[gwrite.plotter]

document_start = '''
G90       ; absolute coordinates
G21       ; mm units
G17       ; xy plane
G00 Z-15  ; pen up
'''

segment_first = '''
G00 Z-20  ; pen down
G00 X{x:.4f} Y{y:.4f}
'''
segment = '''
G00 X{x:.4f} Y{y:.4f}
'''
segment_last = '''
G00 X{x:.4f} Y{y:.4f}
G00 Z-15  ; pen up
'''

document_end = '''
M2        ; program end
'''

unit = "mm"

# scale is (1 / (1/96 inch to mm))
# making 1px in SVG == 1mm in real world
scale_x = 3.779527559
scale_y = 3.779527559

This configuration scales the resulting G-code so 1px in SVG equals 1mm. To plot an SVG file I first convert it to G-code:

vpype --config vpype-config.toml read image.svg gwrite -p plotter image.gcode

Then I start the machine, home it ($H command into grbl), and manually jog the X/Y axis if I want to start the drawing somewhere else rather than in the lower-left corner. Finally I zero-out the axes to where they are G10 L20 P1 X0 Y0 Z0 (so the 0/0/0 is where the tool is), and I'm ready to load up the resulting image.gcode and hit "run".

Useful Links

Anatomy of a CNC Router covers materials and software in way more detail than I did here. PrintNC is the recommended route if you want a DIY project, but not one made completely from scratch.

I've spent about three months lazily noodling on the first iteration of a DIY Pen Plotter. It actually plots:

I collected some learnings and resources from the process here: Building a DIY Pen Plotter.

At Replit, we just released a Desktop App which I helped build. Go check it out and ping me with any issues or feature requests. I also have a couple of other things in the pipeline, if you want to have access to them early, turn on the "explorer" setting in your account.

Other than that, I've spent this quarter lazily working on a DIY Pen Plotter, and treating it as a stepping stone towards making a CNC. The plotter was designed and built completely from scratch, and I'm quite pleased with myself that it, in fact, does plot with a pen:

Adam Wiggins once told me that the best explanations are made by people who just learned something, as they haven't yet internalized that thing and forgotten what was hard about learning it. So, keeping that in mind, I wrote a blog post on building this project.

I also went to the last Strange Loop edition, it was amazing to see so many friends in one place, I had a ton of inspiring conversations, and the talks were great as well. I'm obviously sad that the conference is going away but grateful that we had it for a while.

Worth Checking Out

What I've been reading lately:

• Bringing Design to Software — a great collection of essays edited by Terry Winograd; the book is a bit dated, but still a very interesting read
• Human-Machine Reconfigurations
• How to Solve It, a classic
• Where Mathematics Comes From — Lakoff's take on mathematics origins in metaphors
• Educating The Reflective Practitioner — second part to "The Reflective Practitioner"

On the web:

• Bidirectional Language Note from Elliot
• more typography in code editors
• Jimmy is making a text editor
• Differentiable Programming for Interactive Graphics
• The TTY demystified, and while we're on the topic of shells and terminals, a talk on a new terminal emulator written in Zig and another terminal emulator trying out some new interactions
• a talk on Unit — a cool visual programming language with very interesting semantics
• A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts from Tyler
• this generative art project inspired by Calder's sculptures
• Building a String Art Machine

Fin

I'm going to SPLASH in Cascais at the end of October — let me know if you'll be there too, and if not — catch you in a couple of months!

Today marks my three-month anniversary at Replit!

Going from 10-ish person research lab to 100-ish person startup definitely is an adjustment. For one, it's impossible to know about everything that's going on. Two, small details matter in addition to the "big vision". With exploratory research, it's often a good idea to hand-wave at things that would be solvable with additional engineering time, here, you have to do the "additional engineering" too, which changes the pace (and focus) of the work quite a bit. And three: you actually have feedback from real users and the outside world, and while sometimes emotionally difficult, it dictates a real goalpost to go after.

Working at Replit sometimes feels like building an IDE, and sometimes like iterating on a new operating system. I'm excited about how integrated the product can become, about blurring the line between working alone and collaborating, and between developing an app, deploying, and debugging production issues.

We're growing and hiring actively across the board, feel free ping me if you have any questions!

Smaller Projects

I managed to publish two projects since the last newsletter issue that you might have missed:

• Live Coding Livebook — an exploratory work into enhancing Elixir's Livebook with probing, tracing, and live test feedback
• SDFSheets — a nerd-snipe from Avi Bryant, an experiment in combining spreadsheets and signed distance functions for 2D modeling

For both, there's more info in the linked write-ups, and SDFSheets even has a live demo that you can play with!

DIY Plotting

I've also been very slowly noodling on an extremely over-engineered pen plotter, with a hope to later turn it into a CNC (hence the overkill of motors and supports).

I don't really have a need for a pen plotter, or a CNC for that matter, but I missed making things with my hands (again), and this project is a good excuse to do that.

I'm also doing this completely from scratch, and it's a good excuse to learn about designing more complex machines, and how to use a grown-up CAD software in the process (previously I only used SolveSpace, which is great, but not enough for a project like this one).

So far I'm almost done with the first pass at the design, and starting to print initial test parts to see if they actually fit (nothing ever does on the first try). I'm hopeful that this thing will be able to draw something with an actual pen by the end of the year.

Worth Checking Out

What I've been reading lately:

• two different perspectives on early computing history:
  • What the Dormouse Said
  • A People's History of Computing in the United States
• activity theory and its overlaps with HCI:
  • Mind in Society
  • Context and Consciousness
• and a bit on media theory: Amusing Ourselves to Death, and I'm in the middle of Conscientious Objections

On the web:

• Positional Control in Node-Based Programming
• a great (as always) in-depth tutorial from Amit on making objects draggable
• You Can’t Predict a Game of Pinball
• Tyler has a cool paper coming up, here's a preview
• a cool exploration of movement in visual programming — some Hest and Crosscut vibes
• Supporting Programming with Interactive, Contextual, Structured, and Visual Logs and Multi-level Visualization of Program Execution from the same group

Fin

Have a great summer (touch some grass!), and catch you in a couple of months.

I collaborated with José on a set of research prototypes that show a direction towards expanding Livebook with additional live-coding concepts: probing, tracing, and live test feedback.

None of these ideas are necessarily "new", but then again, what is. For prior art, make sure to check out:

• Usable Live Programming
• Example-Based Live Programming for Everyone
• Babylonian-style Programming
• Projection Boxes
• Margin Notes

Finally, some of the ideas were prototyped in JavaScript and not Elixir, mainly to optimize for research velocity (as in, I'm just faster in that world); there's nothing conceptually stopping these ideas from being applied to Elixir, and some of the prototypes are already being implemented in Livebook which is exciting to see!

Probing

Elixir already has a concept of a dbg() which logs the value out, and passes it through (which already is an improvement over something like print() that requires restructuring your code around it). Since we have control over both the editing environment and the runtime, we can easily extend the editor with live value display:

A couple of things to notice in the screenshot above:

1. adding dbg() calls doesn't change any logic about your program, and can be injected anywhere without having to restructure anything
2. we support inline logging not only for plain values, but also objects and arrays — these can grow huge, so we truncate them after a couple of characters, and you can always get to full view by hovering over them (both with mouse and a cursor)
3. having this sort of always-on visualizations of relevant parts of the code makes it very easy to spot issues — it's easy to see that I forgot to rest.join() to get back to a string from an array of letters

Tracing

Probing described above provides a view into a static "snapshot" of the execution — notice how the dbg() inside map callback only logs out l and we don't see the e from the first loop. Tracing is a way to display probed values over time and see the history of their execution.

Here's an example trace output from processing an array of todos and grouping them by attached tag:

To condense the view, but still keep the system generic, we're displaying "semantic miniatures" of the values. Basic plain values can be easily displayed inline, but complex ones (like objects and arrays), are condensed to display how much they have changed between successive calls — for example +2 ~1 -0 means that two new fields have been added, one was modified, and none were removed. Custom data types can be also provided by the users of the system, building up on the Kino system for defining interactive widgets, already used widely in Livebook.

The programmer can construct a more detailed ad-hoc view by clicking on these values to toggle their full display on/off:

Tracing could be implemented within Livebook as an additional output tab below a cell:

Testing

Elixir provides a way to inline tests inside the documentation strings: doctests. This feature is widely used by both Elixir core and libraries, and Livebook also has basic support for handling them.

As a part of this research project, we explored augmenting them more with live status updates, and integration with the probing and tracing described above.

Each test is enhanced with a traffic light-like status and a random emoji, which acts as an identifier. The tabs below a cell, in addition to output and main trace, also include tracing-per-test, so the programmer can focus on a specific issue one at-a-time.

What feels alive?

Finally, even though it might be obvious, it's important to mention how much better probing and tracing mechanisms work, when they evaluate on every keystroke, compared to being an action you have to take explicitly.

As a programmer, I can stay at my keyboard and just type, and get inline live feedback of how my code behaves. If something is not quite right, it's easy to "wiggle" some structures/values, and see the rest of the code respond ("wobble"), I believe that this is crucial for building such a system that feels good!

This is something that was also an important realization in the Programmable Ink research at Ink&Switch, I think best verbalized here — immediate feedback makes it feel as if you are directly Working With The Material, helps build intuition and leverages Peripheral Vision for spotting patterns and issues.

This project was a result of a nerd snipe from Avi Bryant who worked on a pretty similar exploration called Loglo.

If you want to play around with SDFSheets, check out the live demo.

This prototype provides a couple of very basic tools — a circle and a box, a way to translate and rotate the SDF, and CSG-like operations: union, subtract, and intersect:

You can also build your own SDFs from scratch by calling sdf() with a custom callback:

As usual, this is just a prototype, and there's a bunch of obvious improvements that could be made:

• the usual spreadsheet niceties:
  • click on another cell to insert a reference
  • smart references that work with copy-paste
  • referencing other cells by B2 instead of $("B2")
• support for ranges, so we can sdf.union(B2:B5)
• autocomplete for the code input

As you might remember from my last newsletter, I started looking for the next thing after Ink&Switch at the beginning of the year. I'm happy to report that I'm going to continue working on Future Of Coding efforts as a Design Engineer at Replit. I'm excited about the opportunity to work on these sorts of problems, and at the same time be so much closer to authentic users, the industry, and work on a great product — exactly what I was looking for!

Because January and February were dominated by the interview calls, and in March, I took some time off to recharge, I don't have anything else to share other than notes from the process itself, which you'll find below. I managed to squeeze in a couple of hours here and there on a consulting project for Livebook, but I'll leave that for another time.

Musings from the Job Hunt

Firstly, a huge thank you! to everyone who chatted with me, to friends who connected me to other friends, and to my partner's support through the whole process.

Secondly, this is not a generic guide on how to find a job, but rather observations from my personal experiences. As with every advice, a lot of it only applies to the person who gives it — so here are some unusual aspects to take into account:

• This was my first time explicitly looking to join a company, and be a part of a bigger team. Previously, I was mostly independent, and I joined Ink&Switch through a set of opportunistic coincidences (words which could describe my whole career really) — not through a process like this one.
• I have an unusual set of experiences (and over a decade of them under my belt), and I'm also pretty weirdly specialized — a mix of researcher/designer/developer in the field of complex/computational interfaces is a quite uncommon thing.
• I also built up some online presence over the years, with a lot of public projects and writings, which definitely helped with getting my foot in the door. About half of the interviews I've gotten were inbound, and the other half came from reaching out through my network.
• I'm in central/eastern Europe, which adds an additional dimension to navigate through the process: is the company thinking about me in terms of "cheap outsourcing" or actually needs my skill-set and doesn't care where I am? (the red flag was someone mentioning that I'm in Poland one time too many — fortunately this happened only once)

Thirdly, I chose one of the worst possible times to look for a job, as most of the tech companies were laying off huge percentages of their workforce. Luckily, I only felt this once in my whole interview journey, when a company that I was getting excited about had frozen all their hiring during the process. Also, the phrase "in the current economy" had an appearance on a couple of calls. Overall, I'm very grateful for not having to think about this too much.

With all that out of the way: through January and February I've spent 63 hours on 75 calls (not including the preparation time that easily doubles this), decided to go through seven interview loops, resulting in six overlapping offers. This was probably excessive, but: I wanted to be thorough, get a full overview of my situation, and I used these calls to dial-in what it is exactly that I'm looking for (though having some vague sense of that beforehand was very important). For example, I very quickly learned that the sets of problems that people deal with in the tools for thought community (which often means note-taking), are not something that resonates with me a lot, or that there actually is a difference in a role at a Pre-Seed startup compared to a Series C one (who would have thought?). Generally, "have a lot of conversations" is great advice.

Having alternatives as I interviewed was also very helpful for my anxiety — no single call was infinitely important, which made it easier to stay in the flow of conversation, and not feel pressure that things have to work out. Again, a mutual fit was what I looked for, it wasn't always there, and that's ok.

During the first calls, the whole point was to figure out if there's a space for someone like me. Usually, I hit one of three approaches:

• A company hiring only for very compartmentalized roles, or
• a company accepting an unusual/mixed role, or
• a company really looking for someone like me.

The whole high-level theme here was about leaning into, and capitalizing on, the weirdness (Play Your Own Games), so obviously the latter was my preference, and this is where I spent the bulk of my time.

I generally took "we don't really have an interview loop for this specific position" as a good sign, and tried to work with the company to figure out what's the best way to evaluate the potential fit (and again, the evaluation was bidirectional). When describing myself, I leaned into a "designer with strong technical background" rather than "engineer with some design skills". I also discovered that some companies now have a Design Engineer role, which usually was a pretty good fit (and where I ended up in the end).

Most (but not all) of the interview loops I went through included a portfolio review and a design exercise, sometimes also a deep dive into one of my past projects, including a code walkthrough. All of the companies I interviewed with, had me work on a design-engineering problem, and I've got to play around with designing a VPL, a projectional editor, and tightly integrating code deployments with an IDE. The idea wasn't to solve these problems completely, but to show how I work and think about things.

I drew the line at a coding exercise — at this point I tried to debug the process a bit with the company. My thinking was that with my specialization, the other side wouldn't learn anything seeing me write a binary search or parse some JSON from an API, and that this was not the measuring stick to use for what I'm after.

I spent the most time preparing for the behavioral interviews. I grabbed a ton of questions I could find online, from "why are you looking for a new job?" to "describe a conflict with your manager", and tried to really ruminate on them, and write down my thoughts (some of them were great reflection prompts actually). I didn't memorize the answers, but having a couple of points I knew I wanted to talk about in advance was very helpful, instead of coming up with things on the spot.

The only technical preparation I went through was creating a slide deck for the portfolio reviews. A ton of my projects are online, but I wanted to also walk through the process, give additional context, etc. I strategically picked two projects: Crosscut which shows the research-y and team-lead-y side of me, and Glide's Code Components which was the most recent, most production-resembling, solo work.

During the calls, I took notes on everything, and then transcribed them into a single document where I had a section for every company, with dated notes for the calls I made, including whom I talked to. Through this process, I settled on a set of questions that were important to me, and moved towards creating a structure to compare these companies and their offers against each other. The common set of questions included: company size, runway, funding situation, goals for this year and next five years, and their worries (both about where the company is going, and about the possible role we were discussing).

Coming up with a (somewhat arbitrary) timeline on my side helped move things along. I was saying that I wanted to go through the interviews and make my decision by the end of February ...and that's pretty much exactly what happened.

Setting up a Calendly account was crucial. Navigating multiple daily calls during the reasonable overlap I get with PST would be impossible without it. Also, setting the call duration to 45 minutes, and having a 15-minute break between calls worked out great — there was always space to go over time if necessary, but usually gave me time to make fresh tea, and switch mental contexts before jumping from one call into another.

As for the details of the offers, I don't feel comfortable sharing any specific numbers on the world-wide-web, but I can say two things:

• one, that the distance from the lowest to the highest offer was over 2×
• two, that the offers I were getting rarely were correlated with any of the ranges on the job offers for these companies that I could find online (and still, rarely there were any), or with how well I felt the interview process went, or with how excited the company seemed

Doing actual preparation for the process (asking around, and reading up on how to do this properly) paid off, and as cliché as it sounds — it was definitely an important learning experience. Interviewing and selling yourself is a skill, but as with every skill — it's learnable, and improvable.

I'm extremely happy to have signed with Replit, and very grateful for where I've gotten, and how (somewhat) easy (straight-forward?) this whole thing has been (but don't get me wrong, it was exhausting, and it took me two weeks of doing nothing, to get out of a brain fog that it caused).

Links I Found Helpful for the Job Hunting Process

Career advice:

• https://www.kalzumeus.com/2011/10/28/dont-call-yourself-a-programmer/ (a classic)
• https://lethain.com/forty-year-career/
• https://commoncog.com/so-good-they-cant-ignore-you/

Preparation, and non-technical interview questions:

• https://jvns.ca/blog/2014/01/16/what-my-technical-interviews-have-looked-like/
• https://github.com/learn-co-curriculum/bootcamp-prep-research-before-an-interview
• https://github.com/learn-co-curriculum/bootcamp-prep-answering-non-technical-interview-questions
• https://gist.github.com/nonsie/d07ebc343e23e5b5cd544609d1767f93

Reverse-interview questions:

• https://github.com/viraptor/reverse-interview
• https://jvns.ca/blog/2013/12/30/questions-im-asking-in-interviews/
• https://twitter.com/jensenharris/status/988967889330819072

Salaries and negotiation:

• https://www.kalzumeus.com/2012/01/23/salary-negotiation/ (another classic)
• https://jacobian.org/2021/oct/13/tech-salaries-2021/
• https://haseebq.com/my-ten-rules-for-negotiating-a-job-offer/
• https://haseebq.com/how-not-to-bomb-your-offer-negotiation/
• https://candor.co/guides/salary-negotiation
• https://github.com/petermekhaeil/salary-negotiating

What is equity, and how to value it:

• https://www.benkuhn.net/optopt
• https://www.benkuhn.net/options
• https://blog.pragmaticengineer.com/equity-for-software-engineers/
• https://stripe.com/en-pl/guides/equity-for-employees
• https://gist.github.com/yossorion/4965df74fd6da6cdc280ec57e83a202d

Worth Checking Out

As a palate cleanser, I've spent the quarter reading up on (mostly molecular) biology, some highlights below:

• The Machinery of Life — highly recommended, you should get it even if biology doesn't interest you at all, just for the amazing illustrations!
• Quickstart Molecular Biology, and A Computer Scientist's Guide to Cell Biology for a closer look at how cells operate, and how we know how they operate
• On Growth and Form — very dense classic; I enjoyed seeing more of the connection between biological forms and physics
• Entangled Life — a bit pop-science-ish, but very enjoyable
• Evolution in Four Dimensions — not an easy read, but I loved the evolutionary lens on culture

On the web:

• we published the Untangle essay
• history of Visual Basic
• magic that you can do if you control the whole programming stack
• you should follow Folk Computer for a Dynamicland spin-off powered by RFID localization
  • and you should also follow Val Town for tweet-sized JS cloud scripting
• how to clone a VM in 2 seconds
• Will Wright on game interface design
• FPGA diagrams in Excel; and while we're on this topic: a visibility jam
• another code editor that renders like a game; Makepad also does this, but doesn't have a nice write-up to link to
• adventures in writing a Clojure plugin for Sublime Text, including parsing Clojure with Python, and talking to REPLs
• GPT-powered poem-clock; also how does ChatGPT work
  • and you should definitely follow Geoffrey's writings on End-User Programming × LLM — I don't think I fully grasped what's happening yet (at least so far, this newsletter has been artisanally typed out, keystroke by keystroke)

Fin

Let me know if you got some value out of my job-hunting process write-up, and if you have any questions, feel free to email me.

Have a great spring, and see you in a couple of months!

Untangle is a Programmable Ink research project done at Ink&Switch, together with Marcel Goethals, and with engineering help from Mike Kluev.

This project was an exploration of a visual language for describing, and solving, constraint satisfaction problems.

As usual, there's a long essay documenting our goals, decisions, learnings and failures. Enjoy!

Untangle was presented during my 2022 Strange Loop talk.

the last quarter flew by — I was mostly working on finishing documentations of various projects (which you'll see below), and recovering from a busy summer.

I didn't have much time or brain-space for any serious side-projects. I noodled a bit more on the shape grammar stuff (putting a pin in it for now), and hacked together a MIDI CC looper for my Norns, so I can record and replay knob movements for more organically sounding modulations.

Also, I guess I'm on Mastodon now: @szymon_k@mas.to

Write-ups, a Talk, and a Podcast

This quarter has been filled with a lot of Programmable Ink things:

• My Strange Loop talk is now online. I go over the background of why pen & paper is important, why we might want a dynamic version of these tools, and what they could look like.
• I went on the Metamuse podcast together with James Lindenbaum, to talk mainly about Inkbase, but we also touched on some related topics, including meta-reflections on the research process itself.
• We published the Inkbase essay which surprisingly hanged around on the first page of Hacker News for a couple of days.

Finally, I wrote a long article documenting my research into end-user-programmable UI-builder at Glide, which includes a live version of the prototype — go check it out. If you want to see it in action with additional commentary, I recorded a short informal demo in the last newsletter issue.

What's Next?

After four years at Ink&Switch, two of them as a Principal Investigator, I want to turn the dial from pure research, to more industry-oriented work.

My main interest lays in the space of no/low/future-of coding tools, and end-user programming. I believe that there's a ton of untapped potential in computing, and that a lot of it is limited by (broadly understood) interfaces.

I specialize in solving complex design problems, and approaching them from both engineering, and end-user perspectives. I know how to lead a team, and I'm most interested in an IC role — not looking to become a manager.

The section above documents most of my recent work. If you want to see more, you can explore the projects page, or browse through some of my working notes.

Reach out if you know something that might be a good fit: hi@szymonkaliski.com — I'm looking forward to hearing from you!

Worth Checking Out

Books I enjoyed recently:

• Cognitive Science: An Introduction — a pretty good overview of the field of cognitive science
• Being You — the title sounds cheesy, but the book is not: highly recommended overview of current understanding of biological basis for consciousness
• Draw in Order to See — very relevant to the Programmable Ink thread
• How The World Really Works — (thanks Geoffrey for the recommendation!) another very interesting book with a pretty cheesy title: this one explores various areas of current developed world through the lens of fundamental physical measures — things like how much human-labour-time is needed to produce a kilogram of wheat (two seconds!)
• Rules of Play — (thanks Peter for the recommendation!) a great study of what games are, and how to design them

On the web:

• I feel obligated to share another Ink&Switch essay on Dynamic documents as personal software
• An Engine of Improvement for the Social Processes of Science
• Stop Drawing Dead Programs, and a collection of Classic HCI Demos from Jack Rusher
• 3D modeling with spatial queries
• Multi-touch number picker explorations
• Homoiconic spreadsheets, a great talk from Eli Parra featuring a lot of overlaps with Programmable Ink track: spatial queries, unfolding time into space, and Working with the Material

Fin

I hope 2022 was great for you, and as always, wishing you all the best for the upcoming one!

Glide is a no-code platform which allows end-users to build custom applications from spreadsheets. The users start with providing their data, and then assemble the application from a set of available high-level components. The problem here is, that when the use-case doesn't neatly fit into one of the existing components, there's nothing that can be done — the system is closed, and can't be extended by end-users.

The focus of this research project was prototyping a UI-builder that fits into Glide's aesthetics, and provides a way to edit, and re-use, the visual components available in the system.

The project is document with a long write-up at the Glide research page, and below you'll find some more thoughts (and a demo video!) originally published in Q3 2022 newsletter.

Transcluded from Research at Glide

Glide is a no-code tool for building mobile and web applications from spreadsheets — you provide your own data, choose how to render it from a set of high-fidelity components, and you can get to a working application in minutes. This incredible convenience comes at a cost, of course — if you need something outside the set of pre-made components, you are pretty much at Glide's mercy to make new one for you.

The main goal of the project was to prototype a possible future in which technical Glide users have more freedom in how their apps look and behave, while still keeping the convenience of easily assembling ready-made parts. We followed the intuition that at some level, some code, is ok — and the main question is how it's being packaged and composed into bigger wholes, and re-used at different levels of abstraction.

Below you can find a short narrated walkthrough of the prototype that we've built, to give you some sense of what we've made:

There are a couple of things here that I'm excited about:

• creating a space for local developers (after Bonnie Nardi) — power-users who are comfortable working with small pieces of HTML/CSS/JS, and/or assembling complex graphs for other users to re-use
• using the exact same components through nesting in the layout tree, and inside other graphs
• the semantics of multiple inputs and outputs which allow creating components with custom configuration UIs, and packaging related functionalities together
• this is already a part of Glide, but worth repeating ad nauseam: working with real data inside a live system — the changes that you make to your data tables, to your logic, or visual components refresh live and are always visible — you never have to simulate a computer in your head to figure out what values flow through the system

I know that Glide is actively hiring, and if that sounds interesting, feel free to reach out to them directly or let me know if you want an intro!

Inkbase is another Programmable Ink research project done at Ink&Switch, together with James Lindenbaum and Joshua Horowitz during summer and autumn of 2020.

With Inkbase we explored what would it mean to have a hand-drawn ink combined with spreadsheet-like reactivity, all in end-user programmable form.

To learn more about the research, head over here for a long write-up covering everything from philosophy behind hand-drawn marks, to intricacies of the programming model.

Inkbase was presented during my 2022 Strange Loop talk, and at the 2021 LIVE conference.

this is going to be a shorter update.

As you might remember, I spent this quarter doing a research residency at Glide, where I prototyped a programming system for building custom components, focused on composability, re-use, and liveliness. We're working on a longer write-up, and I'll let you know when it's out. In the meantime, I recorded a short walkthrough to give you some sense of where we ended up, that you can check out below.

Most of my free time this quarter went into preparing for my Strange Loop talk about the research I've been doing at Ink&Switch. From what I heard the talk went pretty well though, and I enjoyed all the hallway conversations that it spawned. The video will be out sometime in the upcoming weeks on the Strange Loop YouTube.

With the little free time that I could find, I continued explorations of some ideas that I first started playing around at Gradient Retreat — using graphs to describe vector shapes, and querying them using Datalog. This project is nowhere near being demo-able yet.

Research at Glide

Glide is a no-code tool for building mobile and web applications from spreadsheets — you provide your own data, choose how to render it from a set of high-fidelity components, and you can get to a working application in minutes. This incredible convenience comes at a cost, of course — if you need something outside the set of pre-made components, you are pretty much at Glide's mercy to make new one for you.

The main goal of the project was to prototype a possible future in which technical Glide users have more freedom in how their apps look and behave, while still keeping the convenience of easily assembling ready-made parts. We followed the intuition that at some level, some code, is ok — and the main question is how it's being packaged and composed into bigger wholes, and re-used at different levels of abstraction.

Below you can find a short narrated walkthrough of the prototype that we've built, to give you some sense of what we've made:

There are a couple of things here that I'm excited about:

• creating a space for local developers (after Bonnie Nardi) — power-users who are comfortable working with small pieces of HTML/CSS/JS, and/or assembling complex graphs for other users to re-use
• using the exact same components through nesting in the layout tree, and inside other graphs
• the semantics of multiple inputs and outputs which allow creating components with custom configuration UIs, and packaging related functionalities together
• this is already a part of Glide, but worth repeating ad nauseam: working with real data inside a live system — the changes that you make to your data tables, to your logic, or visual components refresh live and are always visible — you never have to simulate a computer in your head to figure out what values flow through the system

I know that Glide is actively hiring, and if that sounds interesting, feel free to reach out to them directly or let me know if you want an intro!

Worth Checking Out

This quarter wasn't that great for my book-reading. I read a bunch of things, but can recommend only a handful of them:

• Art and Illusion
• Pieces of The Action — memoirs from Vannevar Bush (of radars, Manhattan Project, Memex, etc.) on the importance of individuals and organizations in R&D
• I took a short detour through a couple of management-related books recommended by various friends: Radical Candor, The Making of a Manager, An Elegant Puzzle

On the web:

• Differentiable Programming from Scratch — great overview of various approaches to automatic differentiation with code samples
• My HCI toolbox: Methods for designing and evaluating UIs
• Screens in Screens in Screens and other fun explorations from Lu Wilson
• Little OLEDs in little Lego bricks
• Node-based tool for procedural motion design

Fin

I'm writing this from ORD, waiting for a flight back home after Strange Loop and Ink&Switch get-together.

I'm looking forward to having more free time now that Strange Loop is behind me. I plan to spend my upcoming evenings making music, and noodling on this datalog-vector-shapes thing I mentioned in the intro — unless something more shiny catches my eye, of course.

In the meantime, wishing you a great autumn, and see you around in December (if not sooner)!

thanks everyone for great feedback and discussions on Inkbase and Crosscut — I really enjoy hearing your perspectives on this work, and it's also helpful since chatting with you all forces me to get better at articulating my intuitions.

I spent the last three months at Ink&Switch working on applying the ideas of sketchiness and conversation with some form of other to a stylus-driven logic solver. We're only just starting to write the essay, so expect it sometime later this year. As always, if this sounds up your alley, I'm happy to give you a demo sooner than that. If you're reading this in your inbox, you can reply directly to this email, otherwise you can always reach me at hi@szymonkaliski.com.

I also managed to work on a couple of smaller side-projects: bothering Paul Shen to expose some of Natto internals, building a simple perspective-transforming document camera, and playing around with a new 3D printer.

use-natto

I've been chatting with Paul about how Natto could become useful in existing contexts. One idea we played around with, is embedding Natto canvases into existing web applications, inspired by "Long-awaited live programming" talk from Joshua Horowitz. Different parts of the application could be (and sometimes already are!) made of smaller DSLs, and these DSLs, in turn, could have their own development interfaces. You shouldn't have to adopt a whole new toolchain or an experimental code editor to get some nice directly-manipulatable dev UIs.

Paul quickly put together @nattojs/eval which allows you to execute Natto flows outside of the web interface. Below is a quick narrated demo of working with an external API in Natto, and bringing it into a simple React application:

Natto recently gained multiplayer cursors, and it's fun to imagine collaboratively live-coding parts of your application, and then including them in the main codebase as an asset file (these canvases are just JSON files that could be easily included in a git repo). You can also imagine having a DEBUG flag which swaps parts of your application to be live-Natto-modifiable.

DIY Document Camera

As you probably know, most of the work I've been doing for a while, has been tablet-oriented. The thing that we're constantly experimenting with are the touch-and-stylus interactions. To visualize what's going on, we usually add an overlay showing fingers and stylus positions, but that is just a proxy for actually seeing how the application is used.

To mitigate this, I've built a very simple DIY document camera using:

• a cutting mat with four ArUco markers glued to the corners
• Logitech C920 webcam
• some old lamp arm I had lying around
• a couple of 3D-printed pieces to attach the webcam to the arm

I use the ArUco markers for perspective transformation to fake the camera floating directly above the table. The code itself is just about 300 lines of Python and OpenCV. At first, I tried to prototype this in a browser using OpenCV.js but the performance was pretty bad — which is unsurprising, as the official build is emscripten, not wasm. I've never really tried the unofficial wasm build, and I just defaulted to what everyone recommends: "just use Python".

Here's a short demo (and a sneak peek at the "sketchy logic solver" project I mentioned in the intro):

And since I know the exact ArUco marker sizes, one additional fun experiment was adding measuring capabilities to the tool:

Of course, this is something that Kevin already explored in detail, concluding that the accuracy is not good enough for use in a CAD context — which I agree with (forget being 1MM accurate with 1080P webcam). I'd be curious to try this with a better camera (anyone knows of any inexpensive 4K ones that are not terrible?), but ultimately good calipers probably win here anyway.

While we're on the topic of webcams, I was so happy to stumble upon uvcc which allows you to configure the Logitech-family webcams without using their clunky app.

Finally, I spent an afternoon trying to package this as an .app and failing with both py2app and pyinstaller. In the end, I made an Automator wrapper around a CLI one-liner which works great for my usecase — starting the app directly from Spotlight search.

3D Printing

My cheap 3D printer didn't survive moving apartments — not to mention that it was incredibly fiddly, and required a lot of hand-holding to do anything. Which is ok if you want to play around with a 3D printer, but not so great if you actually want to 3D print things.

My needs for printing various little designs were piling up, and I finally had enough brain-space to pick a new one. I went with Prusa MINI+, partly because I never really print bigger things, and partly because it fits perfectly between my shelves:

I was half-expecting Prusa to be only marginally better than my old printer, but I was pleasantly surprised — after a couple of hours of assembly the only adjusting I had to do was making sure the Z-axis is correctly distanced from the heatbed. After that I don't have to do anything other than hit "print" and wait, which is awesome.

Summer Plans

I'm writing this from a middle of a vacation road-trip from San Francisco to Galiano Island, where I'll spend a week at Gradient Retreat, most probably tinkering with various things.

Once I'm back, I'm going to spend most of my summer as a "Researcher in Residence" at Glide, prototyping a programming environment for building custom UI components that fit with the Glide's aesthetics.
Just a quick reminder — I do consulting, specializing in digital tools and no-/low-/feature-of- computing research, design & development.
Reach out if that sounds interesting.

I'm going back to the US in late September, this time to give a Programmable Ink talk at Strange Loop (which I'm both excited, and nervous about; not sure in which order).
Hope to see some of you there!

Worth Checking Out

Books I enjoyed recently:

• computers, design, Cybernetic conversation with the material:
  • Design Cybernetics
  • The Reflective Practitioner
  • The Sciences of the Artificial
  • Where the Action Is — some tangible computing, a bit of phenomenology and ethnomethodology, what else do you need?
• Metaphors We Live By and Women, Fire, and Dangerous Things for the impact of language on thinking
• Peak on Deliberate Practice
• Coders at Work — especially interviews with Dan Ingalls, and late Joe Armstrong
• Toward a Theory of Instruction
• Art and Computation — I really enjoyed the argument that computational art has to deal with, and "show", computation — and for that it has to not be functional, not ready-to-hand because then it disappears

On the web:

• if you're not fed up with hearing about Crosscut yet, Ivan made a podcast episode discussing how it differs from Hest
• WFC guy strikes again
• "How Authors Enhance the Readability of Formulas using Novel Visual Design Practices"
• this is how websites should look like
• drawing lambda calculus
• non-keyboard programming at Dynamicland
• and while we're on the topic of logic solvers, Kevin worked on a non-"sketchy" one

Fin

Thanks for reading — as always ping me about anything that seems interesting, and in the meantime: have a great summer!

You can read more about the project in the section below, which originally appeared in the Q2 2022 newsletter.

Transcluded from DIY Document Camera

As you probably know, most of the work I've been doing for a while, has been tablet-oriented. The thing that we're constantly experimenting with are the touch-and-stylus interactions. To visualize what's going on, we usually add an overlay showing fingers and stylus positions, but that is just a proxy for actually seeing how the application is used.

To mitigate this, I've built a very simple DIY document camera using:

• a cutting mat with four ArUco markers glued to the corners
• Logitech C920 webcam
• some old lamp arm I had lying around
• a couple of 3D-printed pieces to attach the webcam to the arm

I use the ArUco markers for perspective transformation to fake the camera floating directly above the table. The code itself is just about 300 lines of Python and OpenCV. At first, I tried to prototype this in a browser using OpenCV.js but the performance was pretty bad — which is unsurprising, as the official build is emscripten, not wasm. I've never really tried the unofficial wasm build, and I just defaulted to what everyone recommends: "just use Python".

Here's a short demo (and a sneak peek at the "sketchy logic solver" project I mentioned in the intro):

And since I know the exact ArUco marker sizes, one additional fun experiment was adding measuring capabilities to the tool:

Of course, this is something that Kevin already explored in detail, concluding that the accuracy is not good enough for use in a CAD context — which I agree with (forget being 1MM accurate with 1080P webcam). I'd be curious to try this with a better camera (anyone knows of any inexpensive 4K ones that are not terrible?), but ultimately good calipers probably win here anyway.

While we're on the topic of webcams, I was so happy to stumble upon uvcc which allows you to configure the Logitech-family webcams without using their clunky app.

Finally, I spent an afternoon trying to package this as an .app and failing with both py2app and pyinstaller. In the end, I made an Automator wrapper around a CLI one-liner which works great for my usecase — starting the app directly from Spotlight search.

it's horrific to look at what's happening in Ukraine. It's horrific to imagine what war is like, and we are still in shock that it's happening right next to us, in our neighborhood. I've heard good things about Razom and that's where we've been donating, and I encourage everyone who can, to do the same.

We're incredibly lucky to be able to move over to nerdy computer stuff in the next paragraph.

I don't have that much to share on the side-projects side this time. In January, I built a simple temperature and humidity Wi-Fi sensor for our basement, just to see if the temperatures there don't go too low, and if humidity doesn't ramp up too high. Do I need a better dehumidifier? Doesn't seem like it right now. The project is nothing exciting, just an ESP32, a sensor I had lying around, piped through MQTT to my home-server Raspberry Pi which runs a bunch of automations using Node-RED, including logging the collected data into an sqlite database.

Sketching and Programming

My talk from LIVE2021, about one of our recent Ink&Switch projects, is finally out. You can watch it below, it's only 10 minutes, and then there's a bit of Q&A:

We're still working on the essay — I'll let you all know when it's out for sure! In the meantime, I wanted to mention a couple of interesting learnings from this project.

Sketchiness of tools. Importantly, this is not a matter of leaving hand-drawn marks, though having them helps. This point is more about working with half-formed ideas, and a big part of it is that the Fidelity of The Tool Should be Proportional to The Maturity of The Idea. Basically, computers are great at enforcing rules (Computer Science Positivism), but tools in the earliest stages of thinking, should provide the exact opposite — freedom to poke at the problem from multiple ends, freedom to leave unfinished pieces lying around in your Peripheral Vision, freedom to be messy.

Programming with live, embodied objects. There's a distinctive difference in how it feels to program a system where everything is represented by a visible, embodied, and responsive representations — instead of one where you refer to things by abstract names, and imagine what will happen when the program runs (by playing Simulator in your head). Importantly, this is not what most of "Visual Programming" (that I know of) tries to solve — these projects are usually about visualizing the structure of the program which is not as useful as Visualizing the Domain of the Problem.

Spatial messages as a way to communicate between embodied objects. The way that most programming was done in Inkbase was by sketching out some shapes on the canvas, and making them react to each other, by asking "what's to the right of me?", "is there anything inside that region?", etc. For example, in the demo below, the pink "wires" are not what makes the logic gates talk to each other — they are there just for user feedback. Instead, the logic gates query for what's to the left (inputs), and send the calculated values to everything that's to the right:

Mixing reactive data-flow and imperative mutations in the same environment. Our objects were programmed using ad-hoc lisp implementation, with a small library for push-based reactivity. The basic API was as follows:

• Reactive chains were set up with ?.
• Imperatively updating values was done with !.
• You could also update a value by piping it through a lambda with ..

Here's an example to make it bit more concrete:

(! value 5)
;; ↑ cell named "value" holds `5`

(! double-value (* (? value) 2))
;; ↑ cell "double-value" holds `10`,
;; and will update as "value" updates

(! logger (log (? double-value)))
;; ↑ `10` logged out to console

(! value 7)
;; ↑ `14` logged out to console

(! ticker (interval 1000 #(. value inc)))
;; ↑ `16`, `18`, `20`... logged out every second

The reality was a bit more complicated, but you hopefully get the idea from this short gist. I think this could be a pretty useful part of the project to turn into a clojure library, but I'd be surprised if there wasn't anything that does this already. I haven't dug around yet, so send me a message if you know of any!

Overall, the project taught me a ton, and charted a direction for a next one, that is also finished, and also, as of yet, unpublished. We managed to push the essay out a couple of days ago after all! The project is called Crosscut, and you can read about it here. The essay is much more comprehensive than anything I would write on my own, and I encourage you to check it out — and do so on a desktop browser as we have a bunch of live demos in there.

Worth Checking Out

Books I enjoyed recently:

• How Buildings Learn: What Happens After They're Built — I really enjoyed the "low road" / "high road" framing for how buildings are used, and adapted, over time; architecture always seemed to me like something "set in stone", but Brand argues that it is far from that
• Visual Thinking — felt a bit dated and dense, but makes a great argument on how all thinking is based in perception
• The Story of Art — great overview of how art evolved from the earliest cave paintings to modernity
• I went on a little (mostly graphical) design thread:
  • Designing Programmes — a cult classic in some circles, somehow didn't make that much of an impact on me, maybe because of my humble "creative coding" background where making programs to make designs is the bread-and-butter of the field
  • Interaction of Color
  • Thinking With Type — this one was excellent, I loved how it plays with the "form", and reminded me a bit of "Understanding Comics" — using typefaces to talk about type, in a similar way as using comic strips to talk about comics
  • A *New* Program for Graphic Design — didn't go into as many details as I wished it would but still a great overview of the field
  • Speculative Everything

On the web:

• Derivability in internal tools — there's something interesting in here, but I can't quite put my finger on it; feels like part of tool design is figuring out what can be done in the tool-space, and what should be pushed "outside"

• (almost) Myst in Figma

• PRIME — a 3D node&wire VPL

• Natto (another node&wire VPL) now has React state boxes — I'm a huge fan of making data tangible and visible, and this is a step in the right direction

• Ethnographic research on Dynamicland

• Tangible 3D Modeling

• Geoffrey is working on moving all of your app state into SQLite — I've built some apps this way with a ton of duct-tape, and I'm excited to see where a more thought-out approach will go!

Fin

That's it for this update, as always feel free to reach out if anything in here is interesting.

See you in a couple of months!

Crosscut is a research project developed at Ink&Switch in the Programmable Ink thread together with Marcel Goethals and Ivan Reese. It is a spiritual continuation of our work on End-User Programming (with an essay here), and the Inkbase project.

As a teaser, with Crosscut you can build-to-explore things ranging from custom step sequencers:

To bookshelf-designing tools:

To learn more, check out the 9000-word essay covering everything from philosophy, through design goals, to the actual implementation of the prototype.

Crosscut was also presented during my 2022 Strange Loop talk if you want to see it in action.

This quarter our family grew with this little guy:

His name is Pablo and he's a boxer.

Juggling finishing renovation of our new place, and potty training a puppy was as hard as it sounds, but we're so happy to have him around.

The rest of the newsletter is a quick update on what I've been up to work-wise, and short description of two side-projects — a MIDI→CV converter and previously unpublished experimental web browser for rabbit-holing.

Starting with work, we just finished another Ink&Switch project exploring stylus-based "programming". We hope to publish soon, but we also hoped that for the previous one, and it's still not out there. Sorry.

As a small aside (slash ramble), "programming" is in quotes, because I really don't like that word for describing this sort of work. I have some early illegible thoughts about this in End-User Programming vs Programming note, but the gist of it is that what I've been after in this research track is "programming" as a tool for understanding, and working with, dynamic systems — same way as a numerals and mathematical notation are tools for working with mathematical ideas. It now seems to me that a lot of Future of Coding work (including most of my own!), is about making lives of "expert symbol manipulators" easier — which is definitely worthwhile — but this year I started to understand that there's a lot of value in making "working with dynamic systems" accessible to everyone, and that might need another approach than iterating from where we are right now.

In any case, I'm excited to share this new project with you, but it will take a while to finalize the paper. If this sounds like something you'd like to get a preview of, send me a message, and we'll set up a call.

MIDI→CV

I'm trying really hard to not get into Eurorack as it looks like a great way to quickly spend a lot of money. I really like a lot of ideas around them — the standardized form-factor, the analog 5V signal as an "API", the physicality of the modules, and the open-endedness of these systems.

Relatedly, one of my favourite effects is Cloudy, an infamous Clouds module in a guitar pedal form-factor. Aside from fitting well into my existing pedalboard, Cloudy exposes some CV inputs, and I wouldn't be myself if I didn't do anything with them.

A couple of afternoons of soldering (MCP4728 DAC and A-Star 32U4 Micro board) and light programming later, and I have a small tool for converting MIDI CC messages into a 0-5V signal. Of course, I'm loosing a lot of the resolution by upscaling 0-127 to the analog range, but that doesn't matter much here. It so fun being able to add additional movement and modulation into the sounds that Cloudy makes, and with eight outputs I have some room for growth here.

Cartographist

During the summer of 2020 I played around with an idea for research-focused web browser, embodying some of the concepts from Browsing vs Searching note — browsing being an open-ended divergent activity, and searching understood as information retrieval.

I shared a preview on Twitter, to a surprisingly overwhelming response, but I got distracted with other things and never got back to the project. I still occasionally get requests for sharing this, so here it is: szymonkaliski/Cartographist.

The main idea of browsing in panes was inspired by Andy Matuschak's website layout and some of Nate Parrot's experiments around stacking mobile web browser views next to each other (which I can't seem to find anymore, sorry). This sort of layout has a long history, starting with Miller columns and the original Smalltalk class browser, and is a great interface for detail-in-context browsing.

As an aside, I also use this technique for navigating code with Vim, where a single shortcut goes to a definition of a function in a new pane:

(Glamorous Toolkit is worth checking out, as it takes the idea of pane browsing to another level)

In theory, I also really like the idea of disk-persisted history which allows for going back to a browsing session after a while, and consciously deciding which "topic" I am in.

Unfortunately, in practice, I don't think having the full history is that useful. Yes, it's sometimes good to know how you ended up somewhere, but I think what's most valuable about "research" is the synthesis part — grabbing parts of larger wholes, rearranging, recombining, thinking with the material. A small step in this direction could be persisting scroll position or maybe selection, and making the history editable — allowing users to remove dead ends, add notes, etc.

Additionally, it started to feel that I'm solving this problem on a wrong level. For example, a good window manager could replace Cartographist almost completely — I played around with columnar layout in HHTWM for a bit, but lack of horizontal scrolling makes it not that useful in the end.

Well, let me know if you have any ideas how to make Cartographist better! Is there anything interesting here that I'm not seeing? Could it be useful to you in any way?

Worth Checking Out

Books I enjoyed recently:

• Digital by Design — a really cool interactive art album
• Rewiring America — a great take on how to stop our carbon emissions with currently existing technologies
• In Over Our Heads — I read this as a way to better understand the context of In The Cells of the Eggplant
• The Embodied Mind — this book was a riot, highly recommended! — I loved combining ideas from Buddhism (like mindfulness meditation), and phenomenology as tools for understanding human experience (hat tip to Marcel Goethals)
• I had fun doing Syntopic Reading on philosophy of science:
  • What Science Is and How It Really Works
  • Against Method
  • The Scientific Method
  • Creativity in Science

On the web:

• Kevin Lynagh has been on a roll lately, researching DIY microfluidics and building a PCB stepper motor
• two new cool visual programming languages: Unit and Circles
• great interactive explainer of Curves and Surfaces
• Peter van Hardenberg (current head of Ink&Switch) appeared on the Muse podcast — he is much better at articulating what we do at Ink&Switch than I am, so it's worth checking out if you're curious

Fin

Thanks everyone for all the thoughtful discussions and great collaborations this year.

Wishing you all the best for 2022!

• Instead of opening new windows (with cmd-click), Cartographist spawns horizontally scrollable panes.
• Instead of forcing you to find things in a linear history, Cartographist shows a tree-structured outline of your browsing:
• Instead of always starting fresh, Cartographist can save, and load "trails" — the exact state of the session you've left — supporting researching topics over long periods of time.

For more context about the project, you can check out the longer write-up below, which originally appeared as a part of Q4 2021 newsletter.

Transcluded from Cartographist

During the summer of 2020 I played around with an idea for research-focused web browser, embodying some of the concepts from Browsing vs Searching note — browsing being an open-ended divergent activity, and searching understood as information retrieval.

I shared a preview on Twitter, to a surprisingly overwhelming response, but I got distracted with other things and never got back to the project. I still occasionally get requests for sharing this, so here it is: szymonkaliski/Cartographist.

The main idea of browsing in panes was inspired by Andy Matuschak's website layout and some of Nate Parrot's experiments around stacking mobile web browser views next to each other (which I can't seem to find anymore, sorry). This sort of layout has a long history, starting with Miller columns and the original Smalltalk class browser, and is a great interface for detail-in-context browsing.

As an aside, I also use this technique for navigating code with Vim, where a single shortcut goes to a definition of a function in a new pane:

(Glamorous Toolkit is worth checking out, as it takes the idea of pane browsing to another level)

In theory, I also really like the idea of disk-persisted history which allows for going back to a browsing session after a while, and consciously deciding which "topic" I am in.

Unfortunately, in practice, I don't think having the full history is that useful. Yes, it's sometimes good to know how you ended up somewhere, but I think what's most valuable about "research" is the synthesis part — grabbing parts of larger wholes, rearranging, recombining, thinking with the material. A small step in this direction could be persisting scroll position or maybe selection, and making the history editable — allowing users to remove dead ends, add notes, etc.

Additionally, it started to feel that I'm solving this problem on a wrong level. For example, a good window manager could replace Cartographist almost completely — I played around with columnar layout in HHTWM for a bit, but lack of horizontal scrolling makes it not that useful in the end.

Well, let me know if you have any ideas how to make Cartographist better! Is there anything interesting here that I'm not seeing? Could it be useful to you in any way?

Code is open sourced: szymonkaliski/Cartographist.

The big "real-life" project which I mentioned briefly in the last newsletter kinda concluded. We're now, finally, living in our new apartment, after 18 months of renovation and construction work, and three years after buying it (in a pretty desolated state). There's still so much to do, but that's how renovations go. At least so I've been told.

It was a long, tiring and frustrating process, but so totally worth it.

On the work side, we finished a programmable ink project at Ink&Switch. We're going to present it during the LIVE 2021 workshop at SPLASH, and we're still working on the final essay which should be out later this year.

We also started ramping up to another project in the End-User Programming track; there even was a job posting, which you could see, if you follow me on Twitter.

The rest of this newsletter is about my new album, and an experimental interface for collection browsing.

Gaze Into The Sun

A bit of backstory first.

Making music was what originally led me to learn programming. When I got interested in making experimental music, Ableton Live was the only thing I had — but everything I was making with it, sounded like, well, Ableton Live. Luckily just around that time Max for Live was added — a way to embed Max/MSP code inside Ableton Live devices. I started re-building various audio effects with my own quirks — a delay with miscalculated time (so it would "drift" a bit), a looper with tons of weird options (and a lot of audio cracks when the loop was readjusted), etc.

Building my own tools was a big part of recording my first album: Out Of Forgetting, released on the Audiobulb label in 2010. Since then, I released an EP, and another full length album, and then nothing happened for a long while.

I played a couple of shows every year, and released single tracks and remixes here and there over the last ten years, but I couldn't finish (or even get to the middle really) any new bigger audio work. I've been collecting material for what feels like forever, and when the pandemic hit last year, and everything slowed down, I also had more brain-space to focus on music.

I spent almost every weekend working on new, and old, tracks. I think the process was also kinda therapeutic for me, or maybe self-pacifying in a sense, I'm not sure.

Once things felt complete, I decided on a whim to share the material with a label that I've been a fan of for a while: Fluid Audio. Dan replied just a day later, and a couple of months after that — here we are — my new full-length album, first one in almost ten years, is available for listening!

I don't think there are any physical copies available anymore, but you can always get the digital version.

Collection Browsing

Reference material and prior art is an important part of research and design work — and since research and design is what I now spend most of my time on, it's important to me to deal with all the different snippets I keep on collecting. I use two online platforms for storing things for later: Pinterest where my main visual references end up, and Pinboard for links to articles and interesting websites. I also take a lot of screenshots and store them all, permanently, in a folder on my hard drive.

These materials are important for my work, and I was worried that since online services tend to change and disappear over time, I might will lose them sooner or later, so a while ago I built a tool for backing all of this up called Archivist.

One problem with the original UI to Archivist is that there's no way to see everything, all at once. I can either scroll and see a couple of images at the same time, or remember the right phrase to search for — this is the main difference of Browsing vs Searching.

You search for a specific thing, and you browse in an open-ended fashion.

I wanted to browse my Archivist collections.

In general, I'm pretty sceptical about UMAP and other ML-y things. I played around with a couple of online "browsers", but they are often used for artificial data (usually the infamous Iris flower data set), or for collections that I didn't make myself, so I have little emotional attachment to them. But, the ability to zoom out to see the whole collection was very compelling, so I decided to try this on my own data.

Archivist Browser (obviously I'm terrible at naming) is a way to browse Archivist collection using the UMAP embedding combined with a high-performance UI.

You can fluidly zoom through the whole collection:

Since images often occlude each other, there's a fisheye distortion around a mouse cursor, which has been useful enough not to need any more sophisticated ways of dealing with this issue yet:

Finally, you can get to the source of the image you're hovering over:

So far I've been using it to rediscover old materials that I forgot about — UMAP seems to be very good at clustering visually similar items together.

The data-processing pipeline is implemented in node.js using TensorFlow.js and umap-js libraries, and the frontend is written in Rust using Nannou framework.

Worth Checking Out

Books I enjoyed recently:

• Computers as Theatre — an interesting look at user interfaces through the lens of theatre, and how we should focus not only on "thinking" but also "emotions" when designing them.
• Tools for Thought — another history-of-personal-computing book, I think I preferred The Dream Machine for that though.
• Understanding Computers and Cognition — Heidegger's Phenomenology applied to designing computer systems. Not the lightest read out there, but highly recommended!
• Computational Design Thinking — hat tip to Gordon Brander, a great collection of essays on, well, computational design.
• Creative Selection — a lighter read about the design process at Apple. Some ideas there seem to be similar to what I talked about in my On Rigor in Design & Research talk. Being able to design the design process still tickles my mind in funny ways.
• Epistemology: Contemporary Readings — collection of Epistemology writings organised under most important concepts for the field: from perception, through problems of induction, to how knowledge works. This got me interested in the Philosophy of Science, so I grabbed the classic Structure of Scientific Revolutions, and I'm currently in the middle of What Science is and How it really works (thanks Patrick for the recommendation).
• Thinking in Systems — yet another classic, definitely worth a read as an introduction to systems modelling.
• Unflattening — a doctoral dissertation in a form of a comic book, talks about points of view and how having more than one "unflattens" our understanding of things.

On the web:

• MakeSpace (which I helped build) is now Sprout and available for free, check it out!
• Kevin Lynagh on Decarbonization
• Hacker News folk wisdom on visual programming — an analysis of what Hacker News generally has to say about "visual programming"
• Understanding UMAP — which was very helpful for the Archivist Browser project presented above
• DeepUI Vector — four years after the infamous DeepUI Concept Video we're finally getting a glimpse at a new prototype!
• Classic Research in Data Visualization — great list of academic research papers on visualizing data
• 3D "sequencer" for making music

Fin

I still can't believe I'm writing this from a couch of an apartment that we actually own.

Always happy to hear your thoughts, see you around in December!

Most of my time over the last three months went into Ink&Switch, where we were ramping up on a new project, culminating some of the research I've been involved in for the past year. We're also finishing (hopefully!) a big "real-life" project, so there wasn't that much time for side-explorations, but I managed to squeeze in two smaller things that might be interesting to some of you.

At Freezeframe we sent out first alpha release to friends & family, it's exciting to get feedback from other people using our small workspace refrigerator. More info coming soon!

Executing Incomplete Programs

The consensus in the "serious programming" crowd seems to be that working with strict types and strict compilers is a good thing, and as a serious developer, you should learn to read the error messages, and work with the compiler to write your program; and then, once it compiles successfully, it works correctly.

I never learned to be patient, and I much prefer a program that does something, even if it does that thing incorrectly. I'm not alone with this, as a friend pointed out:

A running program, even if not correct, feels closer to working than a program that doesn't run at all

Running incomplete programs with type checking is still an open research topic, one of the most famous examples being Hazel, which not only allows you to run unfinished programs, but can also (partially) type check them! It's definitely worth looking into, but: one — it's a research project, and two — a totally new universe to adopt (which you probably shouldn't do right now, because of the first point).

I was wondering if there's a small step I could take in that direction, with an already existing language. JavaScript maybe isn't an obvious answer here, but it has two rarely used features (some would argue, anti-patterns), that allowed me to hack a quick experiment together:

1. with — which extends the scope with the provided object, for example:

   with (Math) {
     // notice that we don't have to use Math.sin or Math.PI
     const x = sin(2 * PI);
   }
   

2. Proxy — which allows you to intercept and overwrite fundamental operations on a given object, for example:

   const obj = new Proxy({}, {
     get(_, property) {
       return `you requested: ${property}`;
     }
   });
   
   console.log(obj.hello);
   // => you requested: hello
   console.log(obj.world);
   // => you requested: world
   

You can combine these two, in a way where every get or apply returns the same Proxy object, creating infinite chains that never fail. Then, you can put that object in with (...), so any time you call for something that doesn't exist, you will get that Proxy object, which will evaluate to itself. This solution obviously doesn't type check in any way, but it's a fun experience to work on a program that (almost) always runs, even if you didn't finish writing it, and it's interesting to see how powerful vanilla JavaScript can be.

Below is a quick demo of building a small React component with immediate feedback and no _ is not defined errors popping up randomly:

Magic Lenses

The second experiment was inspired by research on Toolglass and Magic Lenses by Bill Buxton et al.

Toolglass widgets are new user interface tools that can appear, as though on a transparent sheet of glass, between an application and a traditional cursor.

I've been interested in taking a stab at implementing a system like this ever since seeing various takes on it from Alex Warth and Patrick Dubroy. Here's how far I've gotten in a couple of afternoons:

I found out a couple of interesting things. First of all, magic lenses are fun to use, and feel distinctively different to what most of the computing is like right now. Because the transformations are pure functions (they don't destroy the underlying material, just like a physical lens wouldn't), you can experiment without any fear of losing the original data. You also often get unexpected results, which is great for Novelty Search.

Not everything is great though. It's often hard to say what's going on with multiple overlapping toolglasses, and if the interaction is best understood with no more than one-two lenses, then it might imply, that some other UI primitive could work just as well (or even better). Additionally, z-order matters for the output (stroke blue → mirror results in different image than mirror → stroke blue), and I couldn't find a nice way to visualize that ordering. Making a side-pane hierarchy UI didn't feel right, and automatically reordering z axis on interaction was even more confusing.

Worth Checking Out

Books I enjoyed recently:

• The New Media Reader — it seems as if I was subconsciously collecting the works combined in this book for past couple of years; also, any book on technology that starts with a short story from Borges is worth checking out; I'm not alone in praising this book, and if you were to pick just one of the things I'm recommending, this should be it
• HCI Remixed — collection of famous HCI practitioner's essays on papers that were most influential for them; the book was a bit uneven for my taste, some writings felt like a summary of the underlying material, where some of them gave me new, interesting points of view
• The Act of Creation — cult classic on where breakthroughs come from; quite long but inviting read, going into a ton of historical tangents all circling around the topic of Where Do Ideas Come From?
  • I also read Art & Fear which is almost an antithesis to "The Act of Creation", stating that art comes from repeatedly "walking the walk", which I think Koestler would agree, is just one of the "magic ingredients"
• three nicely complementary readings around "rationality":
  • The Philosopher's Toolkit — as the title suggests, it's a set of tools for making, thinking about, and evaluating (philosophical) arguments; probably not very useful on its own, but often gives an interesting framing of more and less famous philosophical arguments
  • The Scout Mindset — a bit of a lighter read on why "if only they were rational they would understand" is often the wrong framing of a problem
  • In the Cells of the Eggplant — it's the first time I read work-in-progress book (well, what is available right now); even if only partially written, it presents an interesting take on how to be rational about rationality itself
• in my philosophy thread I made a detour into logic:
  • Logicomix — a very light introduction to the topic through the lens of Bertrand Russel's life, in a form of a comic book
  • A Concise Introduction to Logic — a bit more formal introduction to most topics around formal logic, a good jumping-off point into further reading
  • I skimmed An Introduction to Mathematical Logic which I'll for sure revisit once I want to explore a bit more of the mathematical side of logic
  • "Sets, Logic, Computation", "Incompleteness and Computability", and "Boxes and Diamonds" from Open Logic Project — again partially skimmed those, but I really enjoyed the journey from formal logic, through lambda calculus, to modern computation

On the web:

• Monkey Mind Pong
• Simulating Green Beard Altruism — the channel has a ton of great simulation-driven explainers, which make me think again how useful a general purpose Simulator could be
• a great essay on Computers and Creativity
• visualising compilation passes

Fin

What do you think about this shorter style compared to the previous one?

Always curious to hear your thoughts and comments on any of the topics I mention. And in the meantime — see you in a couple of months!

I'm surprised by how much can happen in just three months; below is an update of what I've been up to, random thoughts, and things worth checking out.

Work

I finished my contract with MakeSpace in February, I can't wait for the project to become public so you can play with it — it's so much fun to meet in a space, bring notes and images with you, and see what others are looking at. The project reminds me a lot of the original Engelbartian vision of collaboration — a multiplayer Canvas for Thinking.

I worked on a couple of things with the team — starting with helping figure out some of the main questions for the product, that would guide the technology choices. Next big one was figuring out the actual engineering needed to bring that vision to the web. I've built four different rendering backend prototypes, so we'd be able to compare the performance and their advantages/disadvantages — from basic DOM rendering, through 2D <canvas>, to WebGL — including tricks like culling the viewport while still using the native browser scrollbars for accessibility. Quick, and quite unsurprising realisation is, that as difficulty grows, we can get more performance out of the web stack (we got to 10.000 items on a 5000x3000px canvas at 60FPS, without going to WASM).

We also had fun in the space itself, one of the experiments with biggest impact-to-implementation-difficulty ratio was a styling block, echoing how customisation was done in the MySpace times — by injecting snippets of global CSS. Task-Oriented End-User Programming at its finest. Since MakeSpace is built on top of CRDT, these changes propagate to all users in a given space, which allowed us to live-code the UI from within MakeSpace itself, or have each users' notes styled differently (by adding a .user-${username} CSS class so it can be targeted with a custom selector).

At flow/control, we finished a bit more conceptual consulting project with Clay. We worked on exploratory research on integrations and enrichments — basically how to make Clay a central point for connecting various web applications through a end-user programmable spreadsheet.

As for what's next? I'm going to spend a couple of months with Ink&Switch as a principal investigator. I'm excited for the opportunity to lead research projects aligned with my personal agenda.

Haze

I've had an urge to make some new music software for a while, and the idea of granular looper has been going through my head since Nótt (which featured a custom granulation mode). I grabbed Midi Fighter Twister and Norns and wrote some Lua and SuperCollider code to make Haze:

I wouldn't be myself if, along the way, I didn't build some helper tools. maiden-remote-repl is an extremely simple remote REPL connection to Maiden (a process that provides Lua REPL on Norns). Combining it with file watch script and rsync allowed me to live reload Haze on the hardware — which sped up the actual work by a lot. It's also fun to see the UI refresh almost immediately after saving a file on my laptop. In my experience, spending time on making the project development feedback loop as short and frictionless as possible always pays out.

If all goes well, I should have some exciting music news to share soon.

Freezeframe

I've been working on Freezeframe together with Jason Yuan and Tyler Angert in hiding for a while now. We decided to tease it a bit in February:

Freezeframe is a way to capture and restore your workspaces — opened applications, windows with tabs and documents (where possible), including their spatial layout. I've wished for something like this to exist for a while, and I used to create various crutches to make parts of it happen: from writing AppleScripts to capture Safari tabs to a markdown file, through shell scripts to open multiple applications based on specific project. I'm excited for a unified interface that replaces all of this for me.

This project grew in part out of my experiments with making a tiling window manager for macOS, and trying to bend the OS to my will with my over-engineered Hammerspoon config. I have at least one friend who would ask if I was "playing around" or doing "actual work" — turns out I was doing both.

We're currently working towards an alpha release.

Personal Experiments

Finally, something a bit less technical, still nerdy, just on another axis. For almost a year now, I've been running "personal experiments" on myself. I actually used to say I'm experimenting with ... for a long time, but it was more of a figure of speech than anything else. My main issues were:

• forgetting that what I'm doing is temporary and that I can change my mind later
• forgetting to systematically review and make decisions based on the "experiments"

Luckily I didn't have to look for new software to solve my problems, but I had to devise an algorithm (an Executable Strategy after Andy Matuschak). I started creating special notes which contain:

• the main idea that I'm experimenting with
• the goals — why am I doing this, what do I hope to achieve/change, what's the ideal outcome
• scheduled checkpoints

With checkpoints, I usually leave some questions for myself, so I can compare my previous thinking with my current one. So far I've "completed" about ten experiments, and have a couple currently running (with these regular newsletters being one of them).

I backtracked on some (like not using an alarm clock in the morning), paused two (because my situation has changed), and overall allowed myself to try, what feels to me as "wilder", things out — like capping the amount of hours I'm allowed to work daily, or not allowing myself on the internet before spending some time on Doing My Own Thinking.

This is yet another application of Self-Cybernetics on myself — making my experiments visible to me, reminding myself to reflect on them, and presenting them to myself in a way that works around my psychology.

Running experiments this way was an experiment in itself (tracked within the system; so meta). I concluded it works for me, for now.

Worth Checking Out

Books I've enjoyed over these last three months:

• The Dream Machine — history of computing, from invention of binary logic, Turing machine and stored procedure, up until the internet; it's probably not the only history of computing book that you should read (I have A People’s History of Computing waiting for me, that everyone recommends) but it was a fun, informative read
• A Small Matter of Programming which would be one of my recommendations for anyone interested in End-User Programming (hat tip to Adam Wiggins)
• How to Take Smart Notes which describes infamous Luhmann's Zettelkasten system in detail, and also touches on interesting topics like Where Do Ideas Come From?
• Tools for Conviviality — super dense but extremely interesting take on why modern tools might not be the best thing ever, I could draw some hand-wavy similarities to Seeing Like a State, so if you enjoyed this one it's worth checking the other one out
• About Face — an overview of the whole field of interaction design, I have a couple of other HCI classics waiting for me, so more to come
• I read The Visual Display Of Quantitative Information a long time ago, and with Tufte's new Seeing With Fresh Eyes, I've decided to grab the whole set — it's worth getting for the typesetting and layout alone, not to mention the actual contents
• I've finally finished A New History of Western Philosophy, taking detours on reading some of the original cited works, and I'm excited to dig into more specific topics next

On the web:

• Systems design explains the world

• Rasmus Andersson is working on a new thing which looks like modern take on Plan 9 (and more)

• How NASA Designed a Helicopter That Could Fly Autonomously on Mars

  We use a cellphone-grade IMU, a laser altimeter (from SparkFun), and a downward-pointing VGA camera for monocular feature tracking.

• A tour of Self

• A redditor is working on a ZUI WM

• What even is code?

Fin

Let me know what you think, and see you in a couple of months!

Haze is a four track live granular looper for Norns, optimised for Midi Fighter Twister, but also usable without it.

The SuperCollider engine is based on Mangl, which itself is based on angl and glut.

Incoming audio is recorded into an 8 second loop (with configurable feedback), and granular engine samples from this loop. Each row represents a single track, and there are three banks of parameters, the UI shows currently active bank together with the parameter names and values.

This project is available on github: szymonkaliski/haze.

For other experiments around hardware/software for experimental music check out: Nótt, DIY Monome, and LoopPI.

End of 2020 seems as good time as any to refresh this newsletter, and start sharing more openly again. My plan for 2021 is to publish four of these, roughly one for every season.

This newsletter is divided into two parts — what I've been up to, and what interesting things I came across this year. I'll try to keep everything brief.

This year was tough and weird for everyone, to say the least. Luckily, everyone here is safe and healthy. Poland has been doing quite well earlier this year and seems to be doing pretty bad right now, especially in terms of share of positive tests.

We've been stuck in the same walls for almost a year now, and living mostly the same day on repeat. But it's ok, my personal take on this is that it's a small price to pay for hopefully much more years when we get to go out into the world again.

Even though the beginning of this year was pretty turbulent, I've been lucky to work on a bunch of interesting projects, in no particular order:

• a new interface for working with huge datasets for ML-generated molecular synthesis paths for MoleculeOne
• bespoke layouting system for ground controls screens for European Space Agency (collaboration with Recoded)
• helping with the research process, and working on new ways of collecting information online at The Browser Company
• R&D work for a new financial modeling spreadsheet at Causal

...and a couple of other things I can't talk about.

I've been also continuing research at Ink&Switch around the topic of "programming by drawing" for a tablet form-factor (trying to bring some of the Dynabook ideas to life). Latest spin-off work includes in-house Lisp with a projectional editor. This work is not public yet, but if you're interested in these topics, I'm happy to give a demo on a call.

I'm currently working with lovely folks at MakeSpace on a new way to be together, and doing R&D work at Clay.

Earlier this year I gave a talk at Intersection Conference on the process of solving open-ended problems, if you have 25 minutes to spare, you can have a look here:

On the personal project side, I published a couple of things, in order of twitter likes:

• multi pane "Research Browser" — this one is currently unpublished, I also don't seem to use it that much, I might just publish it as-is soon
• Archivist — which was technically made in 2019, is something I use a lot, I recently added a way to search through screenshots I take (using OCR) and this had a huge impact on how I collect things, I should probably share an update about it soon
• Protoboard and Liveboard — another projects technically from 2019, an answer to a prompt "what if programming was more like using a breadboard"
• FabFungus — yet another 2019 project finally documented in 2020 — an artwork exploring cellular growth and questions of artificial life; selected in the Best Of Poland section at Digital Cultures festival this year (here's me presenting the work)
• HHANN — a way to doodle on your screen, this is super useful when screen-sharing

I also wrote an article on my note-todo system hybrid, and made a subset of my notes public, if you want to browse through my, even more half-baked, ideas.

Finally, onto something other than myself. One success this year was switching my information diet from mostly web articles, to mostly books (and academic papers). Some of the things I read through this year which I'd recommend:

• Mindstorms by Seymour Papert — which shed more light on why Alan Kay keeps talking about finding ways to help children think better — they grow up to be adults after all
• The Art of Doing Science and Engineering by Richard Hamming (with great foreword from Bret Victor) — about half of the book was way over my head (I need to brush up on my math and physics) but I enjoyed the chapters on creativity, and approach to research
• Seeing Like a State
• The Death and Life of Great American Cities
• The Design of Everyday Things
• Thinking, Fast and Slow
• How to Do Nothing
• Understanding Comics — I'm not a big comic fan, but I can't remember when I had so much fun reading a book
• Seeing Is Forgetting the Name of the Thing One Sees
• Think — a quite light introduction to common themes in Philosophy, which made me want more, so I grabbed A New History of Western Philosophy and I just finished the Ancient Greeks section

Various other links worth checking out:

• Kevin Lynagh's newsletter has been a fun read every month, good starting point would be this entry on moving to Taipei to flee the pandemic
• Andy Matuschak research work on his thinking notes and upcoming spaced repetition system
• Glisp — lisp editor for digital graphics from Baku Hashimoto
• assorted notes from Yoshiki Schmitz, including A Pattern Language for Interfaces
• Metamuse a podcast from Muse — a spin-off project from Ink&Switch
• Shape Computation Lab working on "shape grammars" making visual design resemble rewrite rules — it's hard to explain in text, check out their videos!
• Adam Wiggins wrote great personal manifesto on making computers better
• Semilattice, another interesting take on canvas for thinking
• Alan Kay is pretty active on Quora
• interesting take on interactive sketching in AR
• programming with machines?
• I'm approaching my first decade of professional work, so I've been also reading on various things "career":
  • A Forty Year Career
  • Salary Negotiation, Don't Call Yourself A Programmer, And Other Career Advice (and other writings from Patrick McKenzie)
  • Getting to Yes
  • also thinking of things like taking a sabbatical and time needed for thinking

Friends, colleagues, collaborators — thanks so much for everything, wishing you (and myself) that 2021 is a bit kinder to all of us.

FabFungus is a physical installation inspired by questions of digital life and cellular growth. It was created together with Marek Straszak and Arek Zub, and exhibited for the first time at Posthuman_Data exhibition in Poznań in 2019.

FabFungus is a computational tool for creating digital structures grown out with a simulated process. It is also a manifestation of couple of these forms, brought to physical reality through imperfect 3D printing.

FabFungus Photo

FabFungus (...) is a technologically advanced work annexing the theoretical perspective of medianatures and their potential adaptations in the digital-biological environment. In this design, data is part of a new nature, its physical representation is just one of the possible biological interpretations of the code. We are talking here about redefining what is alive and inanimate, artificial and natural. The work raises the question: how much closer to "nature" and biological definitions are algorithms and "non-human" programming languages than anthropocentric strategies for creating cultural narratives and myths?

— Agnieszka Jelewska, Michał Krawczak

From technical side, FabFungus was a culmination of three personal research projects:

• questions about SDF modeling answered with hiccup-sdf
• high-performance simulation algorithms, still partly unpublished: behaviours-rs
• experiments in creating a medium for quick exploratory work: Protoboard

In 2020 FabFungus was a part of Best of Poland section at Digital Cultures, you can see me talking about it below:

Liveboard is a library built on top of Protoboard, providing a set of high-level functions for live-coding music and visuals.

Thanks to hot module reload the functions can be swapped immediately, without the need to refresh the whole page.

Midi-based nodes can be mixed with visual ones, creating an interesting platform for audio-visual performances.

Liveboard provides a couple of useful modules:

• midi for controlling music software
• metronome and tempo — multiple metronomes could be used at once to create polyrhythms
• out as a basic creation block
• a couple of helpers based around music theory (using the tonal library)

Example Liveboard session:

const board = createBoard();
board.start();

metronome(board, [0, 0]);
tempo(board, [1, 0], 90);
midi(board, [2, 0]);

// `tick` is triggered on every metronome tick
// `play` triggers MIDI note
out(board, [2, 2], ({ tick, play }) => {
  const seq = ["G2", "F2", "E2", "D2"];
  const note = pick(seq, tick);

  play({ note, dur: random(2000, 8000) });
});

In 2019 I got a chance to teach a couple of live-coding workshops using Liveboard. Environments with immediate feedback make a great teaching tools, and we also managed to spend some of the time actually playing around and making music.

This project was inspired by breadboards: they are used for both learning, and experimenting, allow for very fast iterations, and tap into our spatial reasoning.

Protoboard is a frontend library built on thi.ng/hdom. There's a single exposed function called createBoard which fills the screen with a grid. Pieces of code can be attached to that grid by supplying a name, position, size, and a function:

import createBoard from "protoboard";

const board = createBoard();
board.start();

board.add("hello world", [0, 0], [2, 2], () => {
  return ["h1", "hello!"];
});

Nodes on the board can hold internal state, which other nodes can react to, forming a computational DAG:

board.add("count every second", [0, 0], [2, 2] (state) => ({
  init: () => {
    // initialize with 0
    if (state.deref() === undefined) {
      state.reset(0);
    }

    handle = setInterval(() => {
      state.swap(count => count + 1);
    }, 1000);
  },

  render: () => {
    return ["div", `counter value: ${state.deref()}`];
  },

  release: () => {
    clearInterval(handle);
  }
}));

board.add("counter times two", [2, 0], [2, 2], ({ nodes }) => {
  return ["div", nodes["count every second"] * 2]
});

Protoboard nodes by design encapsulate code instead of other nodes, compared to some VPLs where even basic math equations have to use blocks and connections.

Combination of the background grid, and a quick way to build DOM elements, allows for adding pieces of documentation, notes, and inspiration images inline on the board, creating a canvas for programming:

Protoboard isn't without issues though. The need to describe position and size of the node in code seems very clean — everything is defined in the same place, and rendering is a pure function over the input file. I just keep on catching myself trying to move the nodes around and resize them using a mouse. Something I want to try is updating the code file when the nodes are moved or resized.

Another issue grows out of the way that nodes are connected together through global naming. This makes it hard to create reusable pieces of code, and the user is forced to create unnecessary abstractions, like:

const createDebugNode = (pos, size, name) => {
  board.add(`${name} debug`, pos, size, ({ nodes }) => {
    return ["code", JSON.stringify(nodes[name], null, 2)];
  });
}

I believe that writing code in vertical slices on some form of canvas has a lot of benefits, we just don't have the proper interactions and abstractions discovered yet.

HHANN was created to fill a basic need of being able to draw on top of the screen.

Allowing users to doodle on the OS-level turns the whole system into a canvas for thinking.

The tool is especially useful during screen-shared calls, or when quickly annotating designs and documents — doodle on top and take a screenshot.

This project as available on github: szymonkaliski/hhann.

For me, the web is one side of the puzzle, although implemented in the way that leaves some things to be desired (Project Xanadu tried to solve that).

The other part, one that feels missing, is a network for my knowledge.

Origins

I stopped trusting my memory a long time ago, and I hate the feeling of stuff slipping through the cracks. I started to use a dedicated todo app sometime in early college, and discovered GTD methodology soon after. This worked well for a while. I've been a happy Things user for a while, and migrated to TaskPaper format when I felt too constrained by Things assumptions.

I even wrote my own TaskPaper parser, and started scripting my task list: automatically turning @due dates to @today at appropriate day (in combination with cron), archiving @done tasks, or a script that would group tasks by date and print them out, so I could see how my next week is looking.

This worked well for a few years, but over time, my tasks started turning more from implement X, to research Y, think through Z, etc.

Once my TaskPaper had more comments than tasks, I knew that this was not going to work for long.

I was also keeping separate folder of notes, initially using Notational Velocity, which then changed to just a folder of markdown files. Some of the notes were related to projects, some of them not. I never knew if a given piece of information should go in my TaskPaper comments, or my other notes. And then, there were notes for thinking about things, which sometimes spawned new projects. This was a disaster in slow-motion.

Fixing the Mess

Since notes and tasks were overlapping so much, I decided to combine them. I wanted to introduce as little complexity as I could, so writing a web app to store all that data in a database didn't feel right. I needed just a bunch of markdown files, some basic structure, and one thing I couldn't live with — that script to show my tasks for the next few days.

The structure that I currently use looks like this:

• /inbox.md - my first-order everything inbox
• /Projects/ - with sub folders per personal project:
  • /Projects/Archivist/, which usually includes at least index.md and worklog.md
• /Work/ - with the same structure as /Projects/, just with work stuff
• /Journal/ - with single file per date — I keep that file opened the whole day, it serves as a scratchpad, brain dump, way to record reminders, ideas, etc.
• /Calendar/ - with single file per year — used to schedule long-term reminders that don't fit anywhere else
• /Notes/ - personal knowledge base

To deal with upcoming tasks I wrote muninn, a small CLI utility which parses all the files, and organises tasks by date. I use github flavored markdown to store my todos, and extended markdown parser to deal with @due(YYYY‑MM‑DD) dates, in the end a file for this article looks like this:

# Memex Article

- [worklog](<./worklog.md>)

## References

- [memex and workshop vs library](<../Notes/Zettels/memex\ and\ workshop\ vs\ library.md>)
- [self-cybernetics systems](<../Notes/Zettels/self-cybernetics.md>)

## Todo

- [x] rough outline @due(2020-04-01)
- [x] re-read outline @due(2020-04-02)
- [x] article setup @due(2020-04-03)
- [x] initial writing @due(2020-04-03)
- [ ] rough pass at first section @due(2020-04-04)

## Outline

- ...

On Inboxes

I have two conceptual inboxes.

First one, inbox.md, is where the most interesting links end up, together with my random thoughts, waiting for a weekly review. I mainly append to that file through iOS shortcut. I can quickly share a twitter link, or jot a quick note in drafts.

Second one is a bit more convoluted. Sometimes during the day I have, well, thoughts. Some of them don't want to go away, and as a way to temporarily "close the loop" I jot them down in my daily journal. Sometimes an idea keeps coming back, and if that happens, I work on it through multiple days in those daily journals, and once it feels good enough, I move it to the longer-term storage — /Notes/ directory.

Weekly review is an important part of this process.
An inbox is not worth much if it's never cleared.

Right now, usually on Sunday mornings, I check out every collected link, and last week of daily journals. At this stage, I'd say half of the things are ignored, because after a couple of days they just don't matter much anymore.

Some things are migrated to be stored as reference materials, articles end up in pinboard, and images/gifs/short videos in pinterest. I then archive those archives in Archivist so I can be sure I won't lose them even if the online services stop working.

On Worklogs

Keeping worklogs had the biggest positive impact on my work in recent years.

The basic idea is simple — to just jot down whatever I'm working on at the moment. Even better, jot down my assumptions, potential issues, and conclusions.

Sometimes I sketch out an API before I write it, sometimes I draw some boxes and lines, then I take a photo and add as an image to that document. Sometimes I outline a few slides for a talk, or just think through how something could work, without implementing it.

This process leaves a visible trail of what I actually did on a given project, which makes it so much easier to get back to things after a break. It also helps fight against Zeigarnik effect — I don't have to keep thinking about things once I wrote them down.

Worklogs also introduced me to the idea of writing as a thinking tool. Instead of trying to "come up" with a solution in my head, I write/draw through/with my thinking process. Writing is a great metacognition tool, it allows me to see my thinking, and debug the thinking process afterwards.

On Thinking by Writing

This all leads my to the way I store notes.

My main inspiration was the infamous Zettelkasten system. I'm not going to introduce the ideas here — The Zettelkasten Method on LessWrong does a great job of that.

• obviously, everything is stored in files instead of index cards
• I don't use stable id to name the notes, I prefer human-readable ones, even if that means I sometimes have to fix broken links
• I'm not against adding quotes to the note, but I never leave it at a quote alone, I always write my own thoughts next to it

The main premise is there though — notes are short, they usually contain one idea, even if I can't name it yet (this also leads to interesting note titles, like spreadsheets might be a wrong model, or understanding requires effort). Notes are interlinked through direct links, and automatic backlinks (via muninn cli tooling). If I have a bunch of notes which seem related in some way, I create index note to collect them together, enforcing a specific point of view. Single note can be in multiple different indexes that way (connections are usually better than taxonomies).

I edit my thoughts in vim, which allows me to easily view multiple notes at once, and with addition of muninn-vim, to explore backlinks. Typical thinking session looks similar to this:

Once something ends up as a note in my system, it was probably rewritten at least a couple of times, through inboxes and daily notes. I put it there when I'm pretty sure it's not a temporary thing-to-remember, but something that will allow me to make new connections, and grow ideas. I try to not replace already existing notes, but build upon them, even if my understanding and ideas change.

Fin.

Like with most of my projects, this is a very personal one. It fits my way of doing things, and similarly like with Timav, changes the things I do. The latter forces me to spend time doing the things I want to do, by showing me the time tracking data in a specific way, the former makes me think thoughts that I probably wouldn't have thought otherwise.

References

• Andy Matuschak's notes — research project on similar ideas
• Roam Research — web app providing automatically backlinked Zettelkasten-like system
• TiddlyWiki — single-file personal wiki system
• Org Mode — emacs-based plain-text organisation suite

Archivist is a set of tools for archiving and exploring of reference materials.

I spend a lot of time gathering resources and lateral thinking material. Over the years I noticed that stuff tends to disappear from the web, both the original content, and services used to collect it.

Archivist fetches local copies of the source images and frozen copies of the websites, so things wont get lost. I'm trading my local disk space for feeling safe that the content stays with me.

This work was partially prompted by local-first ideas from Ink&Switch.

There's no UI except for the content, and no interaction except for scrolling and searching.

Code is open sourced: szymonkaliski/Archivist.

Nótt contains six independent loops, up to 8 seconds long, including speed control, sublooping, reverse playing, and custom granular mode. It combines my previous hardware looper experiment - LoopPI and my DIY Monome research.

It was inspired by mlr and norns, built on Raspberr Pi, Blokas Pisound and Adafruit NeoTrellis using Chuck and Python.

Detailed usage instructions are available here, and build instructions here.

volume-brush is a volumetric brush implementation, inspired and adapted from toxiclibs.

It's using sparse voxel octree for storing the data, and surface nets for meshing the isosurface into a mesh.

This approach opens a lot of possibilities, for creating complex, organically-looking 3d shapes. The library can be easily combined with particle simulations, agent-based drawings, or for example sampling data from images (technique explored by Karsten Schmidt back in 2018).

For more information and examples go to github: szymonkaliski/volume-brush.

Modeler is a CSG modeling library for React, and a cli helper tool.

It is built from two packages: modeler-csg - the main modeling library, and modeler-cli.

modeler-csg is a custom React reconciler designed to work with react-three-fiber:

const React = require("react");
const ReactDOM = require("react-dom");
const { Canvas } = require("react-three-fiber");
const { Model } = require("modeler-csg");

ReactDOM.render(
  <Canvas>
    <ambientLight color={0x888888} />
    <spotLight position={[0, 10, 10]} />

    <Model>
      <subtract>
        <cube />
        <sphere radius={1.3} />
      </subtract>
    </Model>
  </Canvas>,
  document.getElementById("root")
);

modeler-cli provides a quick live-reloading preview, and offline STL export.

For more information and examples go to github: szymonkaliski/modeler.

• functional creative coding library
• time travel abilities
• livecoding editor
• direct manipulation

It was created during January and February 2019 as a part of bimonthly projects challenge that I'm running this year.

Functional creative coding library

Creating graphics in Dacein is done using single sketch function which accepts a few parameteres:

• size - size of the canvas
• initialState - initial state of the sketch
• update - update function which produces new state
• draw - drawing function which consumes current state

state mutations are only allowed within the update function, and even though the state itself is immutable, it can be modifed with familiar imperative JS API thanks to the immer library.

In addition to providing simple mental model - all changes are done in the update function - this technique also allows for easy implementation of time travel.

This fixes one of the issues I have with languages like Processing, where drawing and updating are often intertwined, and in complex programs it's usually hard to follow where the values come from, and where they are changed.

Drawing happens through a small declarative API, borrowing ideas from React - instead of imperatively accessing the canvas API, the user returns an object representing current state of the drawing. This is actually much simpler to implement for canvas vs DOM, because we can just keep redrawing the screen - there's no reconciliation step that needs to happen. The object returned from draw function is an array of arrays, idea borrowed from hiccup. So, combining all of these ideas together, a simple sketch of pulsating circle looks like this:

sketch({
  size: [600, 600],

  initialState: {
    x: 0,
  },

  update: (state) => {
    state.x++;

    return state;
  },

  draw: (state) => {
    const r = (Math.sin(state.x) + 1) * 100;

    return [
      ["background", { fill: "#ffffff" }],
      ["ellipse", { pos: [300, 300], size: [r, r] }],
    ];
  },
});

Compared to similar sketch implemented in p5, which even though shorter, intertwines (global) state manipulation with producing output:

let x = 0;

function setup() {
  createCanvas(600, 600);
}

function draw() {
  background(255);

  x++;
  const r = (Math.sin(state.x) + 1) * 100;

  ellipse(300, 300, r, r);
}

To deal with user input, the update function accepts second argument, which is an array of events that happened between the last and current frame. This array contains objects with source key (borrowed from DOM event sources: mousedown, mousemove, keydown, etc.), and additional event metadata (mouse position, clicked key, etc.).

This allows us to create predictable, declarative sketches, where the output drawing is a function of state, and events that happened in the system.

It's easy to imagine that reasoning and testing such sketches is much simpler than their imperative counterparts.

This part of Dacein project could potentially leave on it's own, without the rest of the IDE. One of the ideas that I didn't get around to, was to allow users to export standalone sketches, that don't require Dacein to run.

If you're interseted in forking Dacein and ripping this part out, let me know. You should also check out hdom-canvas for a much more developed similar idea created by Karsten Schmidt.

Time travel

Because draw and update parts of sketch are immutable, we can record changes made to the state before each draw call, store them, and then allow users to pause, replay, and scrub through time:

The current state value is also visualised, so it's easier to debug and understand what's happening.

The actual implementation of that feature was pretty straightforward.

1. If the sketch is playing, we keep producing new currentState using immer library:

currentState = immer(currentState, (draft) => sketch.update(draft, events));

2. We also keep a history object, where we store both the state and events (mouse, keyboard, etc.), as well as current index inside of those arrays, which we use for scrubbing:

setHistory((draft) => {
  draft.stateHistory.push(currentState);
  draft.eventsHistory.push(events);

  // we make sure we only keep MAX_HISTORY_LEN elements in the arrays
  while (draft.stateHistory.length > MAX_HISTORY_LEN + 1) {
    draft.stateHistory.shift();
    draft.eventsHistory.shift();
  }

  draft.idx = Math.min(MAX_HISTORY_LEN, draft.idx + 1);
});

3. If the sketch is paused, we're not running that part of the code at all, and read the state and events from the history, only calling the draw function:

currentState = stateHistory[index];

The actual implementation is a bit more complex because of how the sketch container is set up, you can check the relevant parts here and here.

Livecoding

Livecoding can mean different things in different context. For Dacein, by livecoding I understand ability to immediately see code changes on the screen, as well as custom pickers to scrub through numbers and colors without typing.

The hot reload of code is again piggybacking on the immutability. When the user changes the code, we can evaluate it again, and swap current draw and update functions to see the changes on the screen immediately (well, by next frame).

This approach becomes problematic when user makes a change to initialState. I tried to automatically patch-in the new initialState on top of current state but it was finicky and was breaking the history scrubbing, so in the end I decided that any change to how the state object looks would automatically reset the sketch (start from initial values).

Here's how it looks like when modifying code:

And here's how the user can play with constants using just mouse:

I'm doing a bit of not always correct detection of what kind of thing is under a cursor, and if I detect a number or string that looks like a color, I'm displaying a custom picker on top of the editor. That picker returns a new value, which is replaced in code which in turn is re-evaluated, leading to new value on screen. This could be optimised of course with some additional engineering time, but I hope it shows how easy this can be to implement in a text editor.

The last interesting thing implemented here in the livecoding space is custom highlight from result, to the code that produced it:

Implementing this was a bit harder. I'm traversing the sketch AST, looking for everything that looks like a drawing command - that is, an array with first item being a string of one of the draw commands. So ["circle", {}] is a draw command, but [10, 20] is not.

When I find an array like that, I augment the second argument with __meta field which holds lineStart and lineEnd values grabbed from the AST. The relevant code is here.

Knowing the origin of a current drawing is only half of the battle. I needed a way to know what the user is hovering over, and since the library uses canvas, this wasn't as easy as adding custom onMouseOver handlers.

What is possible though, is encoding the items identity in color, picking the current color under the mouse, and working back to get to the __meta field.

We can encode and decode limited (but huge) number of values in color using these two functions:

const encodeInColor = (num) => {
  const hex = num.toString(16).substr(0, 6);
  return `#${leftPad(hex, 6, "0")}`;
};

const decodeFromColor = (hex) => {
  return parseInt(`0x${hex}`);
};

We then create another, offscreen canvas, and draw on it using this encoded color, instead of the one selected by user:

const drawInspector = (state) => {
  let i = 0;

  const operations = sketch.draw(state);

  for (const operation of operations) {
    const [command, args] = operation;

    // replace fill and stroke with index encoded in color
    const inspectorArgs = Object.assign(args, {
      fill: args.fill ? encodeInColor(i) : undefined,
      stroke: args.stroke ? encodeInColor(i) : undefined,
    });

    if (COMMANDS[command]) {
      COMMANDS[command](ctx, inspectorArgs);
    }

    i++;
  }
};

We can then pick the color from where the mouse is:

const onHover = (x, y) => {
  const data = ctx.getImageData(x, y, 1, 1).data.slice(0, 3);
  const hex = Array.from(data)
    .map((n) => leftPad(n.toString(16), 2, "0"))
    .join("");

  return decodeFromColor(hex);
};

And then, once we have the index back, we can grab a relevant item from what sketch draw function returns, and pick out it's __meta value, to highlight the code.

It's a bit of long-winded process, but worth it, as it removes the need to keep a tab on where which piece of graphic comes from — something that in my opinion is unnecessary mental requirement while writing code.

Direct manipulation

Direct manipulation is how most of non-developer software works. If you want to move something in Photoshop, you just drag it over directly on the canvas; if you want to change a volume in Ableton you just drag a slider in the mixer. Same when changing fonts in Word, making slides in Powerpoint, etc.

Unfortunately, with programming we are stuck with indirect manipulation, the infamous cycle of change code, reload, test, change code...

With Dacein I tried to bring a bit of direct manipulation when working with sketches, mainly it shouldn't be required to go back to code, just to move something on the screen. When the sketch is paused, anything that has a pos attribute can be selected, and dragged around (with some caveats).

Dacein tries to come up with new code constants within the draw function that fit where the dragged object should be. This is achieved using unconstrained optimisation (thanks to uncmin from numeric.js).

The optimiser is run on a draw function, and tries to minimise the distance between the mouse and currently dragged object.

For uncmin to work, we need a few things:

• a function to optimise
• the initial set of parameters
• a way to encode how far we are from the target solution

In this case, the function we're optimising is the draw part of the sketch, and the thing we're optimising are the numeric constants within it. Usually premature optimisation is not a good idea, but here I knew that replacing the constants in text, and evaluating the modified code again to get the results, would slow down the process a lot. I wrote another AST transform, that walks through all the numbers in draw, and pulls them out into an array, transforming:

const draw = (state) => {
  const a = 20;

  return [["rect", { pos: [a, 10] }]];
};

Into:

const draw = (state, constants) => {
  const a = constants[0];

  return [["rect", { pos: [a, constants[1]] }]];
};

Basically parametrising the code constants. This is invisible to the user, but allows me to run uncmin more efficiently. The relevant AST transform code is here.

The actual magic itself is a fairly straightforward, short piece of code:

const minimised = uncmin((newConstants) => {
  // "draws" the sketch with current state, and constants provided by uncmin
  const drawCalls = sketch.draw(state, newConstants);

  // simple wrapper to get the position of a draw call with given index
  const position = getPosition(drawCalls, id);

  return dist(position, mousePosition);
}, currentConstants);

// minimised.solution is a new set of constants,
// where the distance between position and mousePosition is closest to zero

With some optimisations this works surprisingly well, and makes me wonder why most of our software is not made this way.

If you're interested in working with constraint optimisation, a good place to start would be cassowary, and for not only coming up with parameters, but actually generating new code, miniKanren is worth taking a look at.

Technical notes

Finally, some smaller technical notes from making Dacein.

eval was obviously a huge part of the project, and working with it safely was important, so the IDE doesn't crash. There are three try {} catch() blocks happening before the running code is actually replaced:

1. all AST transforms are wrapped in try as the code might be malformed, and the parser might fail
2. after that the actual eval is also placed in try to make sure nothing goes wrong
3. lastly, when the sketch() command is executed it first tries to run draw(update(initialState)) in hopes of finding some runtime errors

The relevant code is here.

To make the system a bit more open, I implemented a way to use require, thanks to d3-require and bundle.run. I wanted to allow users to use familiar const _ = require("lodash") calls, but d3-require returns a Promise instead of happening synchronously. I wrote yet another small AST transform that turns this code:

const _ = require("lodash");
const vec2 = require("gl-vec2");

sketch({
  // ...
});

Into:

require("lodash").then((_) => {
  require("gl-vec2").then((vec2) => {
    sketch({
      // ...
    });
  });
});

And that was all that was needed to make this work; everything works on client side, and there's no need for a special server and bundling. You can test it out with the particle system example which uses gl-vec2 for some calculations.

Least importantly, the whole app was written using still fresh React hooks, excluding the actual sketch container, where combining internal state, function calls to parent, and orchestrating requestAnimationFrame was easier for me to be done using React.Component. You can browse this part of code here, it's using a lot of discouraged techniques (like componentWillReceiveProps or shoulComponentUpdate() { return false }), but sometimes workarounds are required to get the proper user experience.

Fin.

Dacein is yet another project that is nowhere near being production ready, but hopefully, shows some interesting ideas, and also shows that it's possible to play with those things in relatively short timeframes. It took about 35 hours to build, working after hours and on the weekends.

Dacein is also the first thing made in my 2019 meta-project of bimonthly projects. Unsurprisingly, it feels much more relaxed that my monthly projects in 2017, and also more focused than the last year where I was just trying to do something every day.

I had a lot of fun putting a few different ideas together, and seeing them work well. There are some leftover lower-hanging fruits, like a future-parity of the sketch library to what 2d canvas exposes and squashing some bugs.

For now, I'm moving to the next project, leaving this one open-sourced: szymonkaliski/dacein, and available online: szymonkaliski.github.io/dacein.

Dacein is an experimental creative coding IDE combining few different ideas together:

• functional creative coding library
• time travel abilities
• livecoding editor
• direct manipulation

It borrows heavily for Processing, p5, and React.

Sketches are created in a functional, declarative style, which allows the system to record state changes, and gives the user ability for easy time travel and livecoding.

Additionally, the user can directly manipulate the sketch and thanks to the magic of unconstrained optimization the system is able to create new code constants:

You can read more about the process and ideas behind making this project in this leanghty article: building dacein.

Code is open sourced: szymonkaliski/dacein.

This, combined with publishing weekly studio diary of music sketches, was both easy enough that I knew I could do it, but also kept me accountable.

But still, I missed longer, more focused projects, so I was very happy to be invited for the residency, where I could spent four weeks of focused work on whatever I wanted.

I spent the whole August 2018 in Spokane, Washington at Laboratory, working on open source creative coding tools.

The project

I had a very rough idea of what I wanted to do when I arrived. Few years ago I was exploring 3D growth, and wanted to continue that idea through building my own tools that would allow me to create organic 3D shapes.

First week was full of learning in the most frustrating way — trying and failing over and over again. I still didn't know what exactly I wanted to do, and was exploring different techniques. I finally settled on using SDFs (signed distance fields) after short research on meshing isosurfaces.

After I settled on a technique, I quickly realised that what I really need is a language that would allow me to create SDF models without the need to touch GLSL.

I also wanted to be able to run them on CPU or GPU, and mesh them with the simplest possible API.

One thing that helped me a lot, was starting from imagining and sketching how the API could work like, and then building it to fit my needs. I would create notes.md file, sketch out a few different approaches with comments, and then try to implement the one that looked the most promising. I was keeping dates in those files as well, it's interesting to read them after some time, here's where I realised that hiccup-like language could work:

## 2018-08-11

Thinking of using hiccup-like syntax for building geometries:

```js
const geom = [
  unionRound,
  { r: 0.02 },
  [
    [sdSphere, { r: 0.1, p: [0.0, 0.2, 0.3] }]
    [sdCube, { p: [0.0, 0.2, 0.3], s: [1.0, 1.0, 1.0] }]
  ]
];
```

And for larger datasets:

```js
const geom = [
  withDataset,
  {
    data: somePositions,
    op: [opUnionRound, { r: 0.02 }],
    mapper: [mapProp, { prop: "p" }, sdSphere, { r: 0.01 }]
  }
];
```

One of important things for me to solve with this system was the ability to use large datasets. Creating GLSL from trees like that is fairly straightforward, but there are limitations of the size of GLSL shader, and number of instructions that can be executed on GPU, and I was hitting it often.

The solution at the time was withDataset instruction that later was changed to map, and from the user standpoint works like MapReduce — one functions maps over all the data, and then second one reduces it into single value. Behind the scenes I'm turning this into floating point data texture uploaded to GPU, and then stepping over it in the shader. This allowed me to radically improve number of data points that I can use with this technique, although it obviously has its own limitations.

Metatools

It was really nice to have enough time to be able to tackle related (but not really) problems that I was hitting while developing the main project. For example, I didn't want to wait for requesting/processing large datasets so I wrote persistent-memo to memoize results of functions where input parameters or function body don't change, and persist it over reloads (either in localStorage or on filesystem).

Another issue that I was hitting, was jumping between different sketches, or running multiple different local server instances.

I solved this by creating sketchbook-cli which provides automatic gallery of screenshots on top of folder of flat javascript sketch files. I could see all the sketches I've explored at the same time, and this sped up my workflow a lot, especially since some of the SDF geometries are expensive to render.

Learnings

I've learned some important lessons as well. First of all, a month of residency sounds like a long time, and I was even planning smaller "downtime" projects, but I've obviously never got to them, I barely had time to finish the main project.

Next time, I would probably aim for either fully research/learning related residency, or one where I know almost everything, and produce artwork. I've tried to combine both, and ended up only building the tools, without ever actually making anything bigger with them.

Fin

All of the tools I developed through the residency are now open sourced:

• szymonkaliski/hiccup-sdf - main tool for modeling with SDFs
• szymonkaliski/persistent-memo - memoization that persists over reloads
• szymonkaliski/sketchbook-cli - sketchbook with gallery
• szymonkaliski/behaviours - experimental declarative behaviour modeling that I explored briefly (work in progress)

I also got a chance to play a few music gigs in the US, where I presented new material that I've been working on together with the studio diary project.

I'd definitely want to go for another residency. As usual, working on one thing brings up ten new ideas, and my notebook is only getting bigger. There's something really nice about being able to set aside everything, and allow myself to work around the clock while exploring and learning.

This for sure is not sustainable long-term, but for a month or two, it allows me to create larger chunks of work and not feel burned out, or as if I were avoiding my responsibilities.

hiccup-sdf is set of open source tools made for creating, displaying and exporting 3d models made with SDFs.

It's another iteration of my exploration of SDFs started with SDF-UI.

This project consists of three libraries:

• hiccup-sdf - main library providing simple language for building models
• display-sdf - utility for displaying SDFs
• hiccup-sdf-to-obj - utility for exporting hiccup-sdf models to OBJs, for 3d printing or other uses

Modeling is done with language inspired by hiccup, adapted to use in javascript:

const { displayRaw } = require("display-sdf");
const { compileShader, glslHelpers } = require("hiccup-sdf");

const translatedSphere = (t, r) => ["translate", { t }, [["sphere", { r }]]];

const shape = [
  "repeatPolar",
  { r: 10 },
  [
    [
      "translate",
      { t: [0.5, 0, 0] },
      [
        [
          "intersection",
          {},
          [
            ["box", { s: [0.1, 0.1, 0.1] }],
            translatedSphere([0.15, 0, 0], 0.2),
          ],
        ],
      ],
    ],
  ],
];

const { inject, model } = compileShader(shape);
const shader = glslHelpers.createShaderFull(model, inject);

displayRaw(shader);

SDFs can be meshed using surface nets and hiccup-sdf-to-obj provides set of tools to help with that, and export 3d-printer ready models:

For more information and examples go to github: szymonkaliski/hiccup-sdf.

This project was developed in part at Laboratory, an artist residency for interactive arts: laboratoryspokane.com. You can read more about my time in Spokane in this blog post: Laboratory Residency.

I've built my own window manager for macOS that gives me layout-based automatic tiling.

Not having to think about placing windows on the screen, and having everything always visible, has been a huge boost for me.

HHTWM is library with small API to be used within larger Hammerspoon configuration.

It's open sourced and well documented: szymonkaliski/hhtwm.

It complimentes live-reloading with live-screenshotting capabilities, providing a gallery with previews generated from code.

The tool was built to help me see all the sketches that I'm working on, as I usually create a bunch of small random files to test ideas and play around.

These files are usually named either 0001.js or 20181029-1814.js, so being able to see what they all output at the same time removes a lot of friction from my sketching workflow.

sketchbook-cli is open sourced: szymonkaliski/sketchbook-cli, and can be installed via npm: npm install -g sketchbook-cli.

Most of the live-reload code was adapted from budo.

This project was developed in part at Laboratory, an artist residency for interactive arts: laboratoryspokane.com. You can read more about my time in Spokane in this blog post: Laboratory Residency.

User has access to new top-level function: def. This function is used to define constants and functions which are live-reloaded, with persisted state (unless the body of the function has changed):

def("a", 10);
def("b", 20);
def("sum", (a, b) => a + b); // sum is "tied" to defs "a" and "b" here

This environment is ideal for quick sketches and explorations. It provides all that observable exposes, including DOM access and more.

File-based notebooks have few additional nice properties:

• they work offline, even if the online serivce goes out of business
• they compose well with standard javascript tools: prettier, eslint, etc.

editable-cli is open sourced: szymonkaliski/editable-cli, and can be installed via npm: npm install -g editable-cli.

def("chart", (DOM, data, margin, d3, yAxis, xAxis, y, x, width, height) => {
  const svg = d3.select(DOM.svg(width, height));

  svg
    .append("g")
    .attr("fill", "#444")
    .selectAll("rect")
    .data(data)
    .enter()
    .append("rect")
    .attr("x", x(0))
    .attr("y", (d) => y(d.name))
    .attr("width", (d) => x(d.value) - x(0))
    .attr("height", y.bandwidth());

  svg
    .append("g")
    .attr("fill", "white")
    .attr("text-anchor", "end")
    .style("font", "12px sans-serif")
    .selectAll("text")
    .data(data)
    .enter()
    .append("text")
    .attr("x", (d) => x(d.value) - 4)
    .attr("y", (d) => y(d.name) + y.bandwidth() / 2)
    .attr("dy", "0.35em")
    .text((d) => d3.format(".3f")(d.value));

  svg.append("g").call(xAxis);
  svg.append("g").call(yAxis);

  return svg.node();
});

def("margin", { top: 30, right: 0, left: 30, bottom: 10 });
def("height", (data, margin) => data.length * 18 + margin.top + margin.bottom);
def("width", 600);

def("alphabet", (require) => require("@observablehq/alphabet"));
def("d3", (require) => require("d3"));

def("yAxis", (d3, margin, y) => (g) => {
  return g
    .attr("transform", `translate(${margin.left},0)`)
    .call(d3.axisLeft(y).tickSizeOuter(0));
});

def("xAxis", (d3, margin, x, width) => (g) => {
  return g
    .attr("transform", `translate(0,${margin.top})`)
    .call(d3.axisTop(x).ticks(width / 80))
    .call((g) => g.select(".domain").remove());
});

def("y", (d3, margin, data, height) => {
  return d3
    .scaleBand()
    .domain(data.map((d) => d.name))
    .range([margin.top, height - margin.bottom])
    .padding(0.1);
});

def("x", (d3, margin, data, width) => {
  return d3
    .scaleLinear()
    .domain([0, d3.max(data, (d) => d.value)])
    .range([margin.left, width - margin.right]);
});

def("data", (alphabet) => {
  return (alphabet || [])
    .slice()
    .sort((a, b) => b.frequency - a.frequency)
    .map(({ letter, frequency }) => ({ name: letter, value: frequency }));
});

I collaborated with Humanitarian OpenStreetMap to design and build an online tool for tracking changes made to OpenStreetMaps map layers.

The tool allows users to start quickly by selecting one of predefined modes (last week, month, year), focusing on relevant viewport, and sharing the output. For experienced users we expose options to play with colors, map styles, the length, and speed of the animation.

Visualization can be shared both as a link to online version, and as a video/gif static export, which makes it ideal for embedding on websites or in presentations.

flow/control was responsible for the design and development of this project, from start to finish. The different parts of the system are containerized and communicate through API calls, and RabbitMQ queue. Offline rendering uses exactly the same code as the frontend, but is executed within offscreen Electron instance, which allowes us to grab each of the frames, and compose them into static video or gif file.

The tool is available online: http://visualize-change.hotosm.org, and the code is open sourced: hotosm/visualize-change

CRLN is an experiment in bending curl noise on the GPU (desktop Chrome only).

Particle movement is calculated on the GPU using framebuffer ping-pong technique.

Movement is controlled by curl noise combined with rotation with varying parameteres that change over time, independently of each other, which creates endless combinations of values.

CRLN was built with THREE.js.

Full source code: szymonkaliski/CRLN.

MNTN is a small experiment in creating procedural mountains.

Mountain generation begins with a 2d plane, that is then deformed by simplex noise.

Deformations are controlled by array of points and distances, which allowed me to have more control over final scene, even if the actual mountain shape is different on every visit.

MNTN was built with using simplex noise, and is rendered with THREE.js.

Full source code: szymonkaliski/MNTN.

I've built two DIY Monome clones as last project of my 2017 one-project-a-month challenge.

More info in the detailed blog post, and full source code and instructions on github: szymonkaliski/diy-monome.

Intro

I remember when the first Monome came out around 2007, a simple (and not necessarily new) idea of grid with 64 buttons and 64 LEDs behind them. It was a beautiful little controller, fully programmable, and I dreamed of getting one.

It was around the time when I started experimenting with max/msp and making my own music, but I've never bought one.

Since then, Monome has created a lot of different devices, from Monome companion — arc, to fully programmable DSP computer (aleph), and recently a lot of modular hardware.

For a while Monome was quite popular in music-making scene, and at some point there was an effort around arduinome project, which provided Arduino code and build instructions for building DIY Monomes. With appearance of Launchpad and other cheap grid-based controllers that project died off.

I kept thinking about getting one of them at some point, and by accident, stumbled upon Sparkfun 4x4 PCB, and later at Adafruid 4x4 PCB and decided to finally build my own.

Building Sparknome

I bought four Sparkfun boards, along with some RGB LEDs and diodes, and started assembling single 4x4 board first. Sparkfun PCBs are build to support scan matrix technique, where each column is queried one by one, reading values of all the rows in that column. If this is done very fast, user has a feeling of near-immediate feedback, and button grid can be driven with relatively few connections. I started by wiring single 4x4 grid, connecting four rows and four columns, testing and rewriting the code provided by Sparkfun.

Next step was connecting four 4x4 grids together to form big 8x8 grid. Unfortunately, Sparkfun didn't prepare the board for this (by, for example, providing paths to wire the boards together like Adafruit does), so I had to do a lot of wiring, and it didn't go too well. I'm pretty sure I could redo it in a much nicer way, but I was rushing to get the prototype working.

Next step was designing, and 3d-printing the case, since the boards were only connected by wires, the whole structure felt very fragile. As the case was a simple rectangle with holes for buttons and screws, I thought it would make sense to design it in code. I already played a with CSG (constructive solid geometry), some time ago while exploring thi.ng clojure libraries, and later while building SDF-UI.

For this project I went with csg.js library combined with three.js for live preview. Rendering required me to figure out how to convert csg.js polygons to three.js geometry:

const geometryFromPolygons = (polygons) => {
  const geometry = new Geometry();

  const getGeometryVertice = (geometry, v) => {
    geometry.vertices.push(new Vector3(v.x, v.y, v.z));
    return geometry.vertices.length - 1;
  };

  for (i = 0; i < polygons.length; i++) {
    let vertices = [];

    for (let j = 0; j < polygons[i].vertices.length; j++) {
      vertices.push(getGeometryVertice(geometry, polygons[i].vertices[j].pos));
    }

    if (vertices[0] === vertices[vertices.length - 1]) {
      vertices.pop();
    }

    for (let j = 2; j < vertices.length; j++) {
      const face = new Face3(
        vertices[0],
        vertices[j - 1],
        vertices[j],
        new Vector3().copy(polygons[i].plane.normal)
      );

      geometry.faces.push(face);
    }
  }

  geometry.computeVertexNormals();
  geometry.computeBoundingBox();

  return geometry;
};

This code basically iterates over all the polygons, adding them to geometry one by one, and remembering the indexes for creating faces. Generating the csg geometries was quite fun, for example, to get the rectangle with button holes, I started with a simple cube:

const genCasing = ({ caseHeight, caseSize }) => {
  return CSG.cube({
    corner1: [0, 0, -caseHeight],
    corner2: [caseSize, caseSize, 0.0],
  });
};

And then cut out the buttons:

let casing = genCasing({
  caseHeight: CASE_HEIGHT,
  caseSize: CASE_SIZE * MONOME_SIZE_MOD,
});

// button holes
let buttons = [];

times(8).forEach((i) =>
  times(8).forEach((j) => {
    const x = i * (BUTTON_SIZE + BUTTON_OFFSET) + BUTTON_CASE_OFFSET;
    const y = j * (BUTTON_SIZE + BUTTON_OFFSET) + BUTTON_CASE_OFFSET;

    const button = CSG.cube({
      corner1: [0, 0, 1],
      corner2: [BUTTON_SIZE, BUTTON_SIZE, -1],
    }).transform(CSG.Matrix4x4.translation([x, y, 0]));

    casing = casing.subtract(button);
  })
);

I also added some debugging information to the preview, so I could see not only the final shape, but also all the "helper" geometries used to cut out things:

Three.js GridHelpers were very helpful in making sure my measurements were correct. I was using a ruler to measure the PCB, and then recreating the shapes in 3D.

Once the case model was done, and I got the STL export working, I spent a few days playing with different Ultimakers Cura options to get the print working on my 3D printer. It was quite a challenge since the exported model was 20cm x 20cm — exactly the size of my printers heated bed. The solution was to disable any brim printing, and disable the printer from avoiding zero distance from the rim of the bed.

Next step was to get this thing to talk through serial in a way that serialoscd understands, so it can communicate with hundreds of max/msp patches available online. Luckily, Monome provides serial reference online: monome.org/docs/serial.txt, and I also found one implementation on github by Mike Barela: Untz_Monome, that I could use as a starting point. To get serialoscd on my system, I used linux instructions for compiling it by hand, which boils down to git clone, ./waf configure and ./waf install. serialoscd is also available on homebrew, but I wanted to have access to the source code, so I could debug it while trying to understand what's happening.

I quickly stumbled upon the first bigger issue, as serialoscd recognises the device as Monome based on FTDI name, I had two solutions, either flash my Arduino FTDI eeprom, or hack around this in serialoscd. I chose the latter, and luckily the change required was single-line update in devices.h:

static monome_devmap_t mapping[] = {
  ...
  /* diy monome */
  {"%d", "mext", {0, 0}, "monome i2c", NO_QUIRKS},
}

This basically recognises any device with some numbers in the name as Monome — this probably would lead to some problems further down the road, but worked for now.

I also had some problems establishing proper "handshake" between Arduino and serialoscd, that I fixed by just increasing the timeout that serialoscd waits for response for the device. I suspect that official monomes are much faster at processing serial messages than the code I wrote for Arduino.

After all of this, I finally got my Sparknome to appear in Monome Home. From there on it was few more hours to get all the messages to properly send both ways, and I had working Monome clone on my hands.

Well, it wasn't working completely, due to my hardware wiring issues, but at that point I already ordered Adafruit based boards, so I decided to build a second one, before deciding if I should work on that wiring.

Building Trellinome

Soldering Trellinome was much quicker, mainly because there was much less to solder — only the LEDs, and few pins to connect the boards together. The important point was shortening the jumpers at the back of the boards, so they can address properly. For Arduino, I went with much, much smaller Arduino Pro Mini (5V version, since the boards are poweresd by 5V).

I slightly modified the code for case generation, and printing was also much simpler because Trellinome is smaller compared to Sparknome.

Overall, I had the board ready and seemingly working in just a few evenings of work, but there was one problem I discovered while testing: if some mext-compatible application flood the serial connection, the serialoscd would hang, and stop sending messages to Arduino. I was pretty sure the Arduino itself wasn't a problem, since key events were still flowing back to the applications, just the display wasn't refreshing.

I spent a few good days trying to debug serialoscd and find the culprint, but in the end I decided to try and write my own replacement.

I went with my familar ecosystem of node, node-serialport and omgosc for this. After getting basic communication with Monome, I spent quite some time trying to understand all the endpoints that serialoscd supports. I set up two different scripts that I used to grab all the messages flying between serialoscd and max/msp patch. I first recorded initial sequence of messages from serialoscd, then re-played them to Monome Home to get next batch of messages, and so on, until I found out what execatly was happening. I could call this "paused" man-in-the-middle attack, but I really wanted to be able to not only see the messages, but also send different ones to understand the replies.

In the end, it took me some time to test my serialoscd implementation with a few different scripts, but I'm happy to say it works with everything that I needed it to, and doesn't hang when flooded with messages.

Fin.

This project was extremely fun, but also took way more time that I anticipated. I started with the idea of building working prototype in first half of December, and then making some music with it, but that music-making part didn't happen until mid January.

Either way I'm very happy with my final Trellinome, it's really nice to play with, but I suspect it's nowhere near the quality of official Monome hardware.

This project also concludes my one-project-a-month experiment, and I'm moving on to different goals in 2018.

I'm going to write some kind of summary of 2017 at some point in the future, but for now, I plan on taking some time off of writing long blog posts.

As usual, the code is open sourced on github: szymonkaliski/diy-monome.

This year, while doing one-project-a-month, my attitude towards this changed, and I realised that buying things that push me forward and make me learn is always worth the money.

Hardware

I found a cheap Prusa i3 clone from FLSUN and ordered the DIY kit for November's project. Assembling the printer was a great fun as I love hardware projects and working with my hands. I've spent a few enjoyable long evenings listening to podcasts and screwing it together.

Parts of instructions were hard to decipher (mostly due to not the greatest English translation) but overall, I managed to build it without any bigger problems. Some of the connecting cables are very short (for example LCD connection to main board), so placing everything neatly on the frame was a bit of a challenge.

Software

One of the biggest problems was connecting the printer to OSX. I've wasted a few hours loading kext's by hand trying to force OSX to see the device, and finally, I stumbled upon this thread on RepRap forum, and this driver for MKS Gen_L v1.0 board that the printer came with.

Instructions attached to the printer recommend use of Repetier for printing, and Slic3r for slicing, but I've found that the recommended configuration setup is at best confusing (there are different settings presented as the recommended ones in different README files).

Luckily, someone on the FLSUN Facebook group suggested Ultimaker Cura as a replacement, since FLSUN i3 is compatible with Prusa i3. Cura was much simpler to set up, and produced better print than combination of Slic3r and Repetier did for me.

Fin.

I'm pretty happy with my printer, and like I hoped, my ideas list grew with a few new entries while assembling and using it.

So far I only made a few test prints including simple spool holder, and I plan on using it for making enclosure for my December project. This will require getting into 3D modeling, and I'm excited to learn!

October was all about exploring different creative-coding-related uses of Haskell: famous live-coding environment — Tidal, and (a bit less) famous graphics library — Diagrams.

Tidal

Tidal is a live-coding environment for creating live music. It's mostly sample-based, using SuperDirt as a backend (which itself is SuperCollider addon).

In Tidal, music is based around patterns — there's nine channels (named d1 to d9) where they can be sent through to SuperCollider. The most basic example is playing single sample every cycle:

d1 $ sound "bd"

Complex patterns, in combination with audio effects, can create interesting small compositions:

d1 $ fast "0.75" $ n "d3? d4*2 e3 a3 f3 c3?" # s "supersaw" # speed "0.3 0.2" # shape 0.6 # room 0.8 # delay 0.4
d2 $ fast "0.25" $ n "a2 ~ a1 ~ c2" # s "supersaw" # speed "-0.1" # room 0.9 # shape 0.9 # gain 0.9
d3 $ fast "1.5" $ n "d7 a7" # s "supersaw" # speed "-0.9" # room 0.2 # shape 0.9 # gain 0.5 # delay 0.8

I played with Tidal during first two weeks of October, going through Tidal tutorial and Tidal patterns documents.

Log of my experiments is available online: szymonkaliski/haskell-playground/tidal.

Tidal, although extremely interesting as stand-alone project, didn't make me feel that I was gaining any Haskell knowledge. It has its own set of functions and uses quite small part of Haskell language.

This is not something good nor bad, I feel that Tidal can be extremely productive and is quick to build up complex patterns of sounds, but after two weeks I wanted to explore more of Haskell itself.

Diagrams

Diagrams is Haskell's DSL for creating vector graphics. I followed the quick start tutorial, and tried to build at least a simple graphic every day, first by making simple shapes, then patterns, and later using noise, and finishing the month with L-System implementation.

One thing I really enjoyed, is how terse Haskell can be. To build this shape:

I only needed this:

blackCircle = circle 0.1 # fc black
circleEdges w = atPoints (trailVertices $ regPoly w 1) $ repeat blackCircle

diagram :: Diagram B
diagram = foldr1 mappend $ map circleEdges [3,6..30]

I wouldn't say that terseness of a language signifies in any way how powerful it is, or that the amount of letters that have to be typed influences productivity in any way but, for me, shorter functions with little syntax, feel easier to read.

L-System

Last part of my two weeks with Diagrams was spent building simple L-System from scratch. I started with basic rule rewriting code:

rules = Map.fromList [('F', "FF+[+F-F-F]-[-F+F+F]")]

getRule axiom = case Map.lookup axiom rules of
                     Just result -> result
                     Nothing -> [axiom]

genGeneration axiom = concat $ map getRule axiom

genGenerations :: Int -> [Char] -> [Char]
genGenerations 0 axiom = axiom
genGenerations n axiom = genGeneration (genGenerations (n - 1) axiom)

• rules is a Map from axiom to some rule, getRule queries that Map and returns matching rule or input axiom if no rule is matching
• genGeneration (no idea what would be a good name for it) generates new axiom based on current one, by concatenating results of getRule
• genGenerations (even worse name) runs genGeneration recursively n times (using awesome Haskell pattern matching)

All this code can be tested in ghci:

*Main> genGeneration "F"
"FF+[+F-F-F]-[-F+F+F]"

*Main> genGenerations 2 "F"
"FF+[+F-F-F]-[-F+F+F]FF+[+F-F-F]-[-F+F+F]+[+FF+[+F-F-F]-[-F+F+F]-FF+[+F-F-F]-[-F+F+F]-FF+[+F-F-F]-[-F+F+F]]-[-FF+[+F-F-F]-[-F+F+F]+FF+[+F-F-F]-[-F+F+F]+FF+[+F-F-F]-[-F+F+F]]"

The next thing that I needed was a turtle, following my L-System rules:

• F - move forward and draw a line
• G - move forward without drawing a line
• + - turn right
• - - turn left
• [ - save current position
• ] - load last saved position

I started by creating some types:

type Angle = Double
type Position = (Double, Double)
data Turtle = Turtle Position Angle deriving Show
type TurtleStatus = (Turtle, [Diagram B], Stack Turtle)

• Angle and Position should be self-explanatory - one is just an alias for Double, another is a tuple of two Double: x/y position
• Turtle contains Position and Angle
• TurtleStatus (another name that could be improved) contains current turtle information, array of Diagrams that it created, and Stack of Turtle informations (for saving and loading positions)

For [ and ] L-System rules I've created simple Stack implementation, so turtle information can be pushed and popped when needed:

data Stack a = Stack [a] deriving Show

push :: a -> Stack a -> Stack a
push x (Stack xs) = Stack (x:xs)

pop :: Stack a -> (a, Stack a)
pop (Stack (x:xs)) = (x, Stack xs)

This way I could execute the turtle over all of the L-System rules, gather Diagrams that it created, and generate graphics in the last step. To move the turtle forward I'm calculating an end point using current turtle position and angle (notice how drawLineAndMoveForward returns a Diagram):

moveForward :: TurtleStatus -> TurtleStatus
moveForward ((Turtle (x, y) angle), ds, st) = ((Turtle (nx, ny) angle), ds, st)
  where nx = (x + (sin (angle / 360) * pi) * stepDistance)
        ny = (y + (cos (angle / 360) * pi) * stepDistance)

drawLineAndMoveForward :: TurtleStatus -> TurtleStatus
drawLineAndMoveForward (t, ds, st) = (nt, d:ds, nst)
  where (nt, _, nst) = moveForward (t, ds, st)
        d = fromVertices $ map p2 [(getPosition t), (getPosition nt)]

Next step is to get turtle function for given rule, and execute set of rules:

getRuleFn :: Char -> (TurtleStatus -> TurtleStatus)
getRuleFn r = case r of
                   'F' -> drawLineAndMoveForward
                   'G' -> moveForward
                   '+' -> turnRight
                   '-' -> turnLeft
                   '[' -> saveLocation
                   ']' -> restoreLocation

executeTurtleRules :: [Char] -> TurtleStatus -> TurtleStatus
executeTurtleRules (r:rs) t = executeTurtleRules rs $ getRuleFn r t
executeTurtleRules [] t = t

To get the final diagram, we have to execute turtle rules, get the diagrams part of TurtleStatus and fold them into single diagram:

getDiagrams :: TurtleStatus -> [Diagram B]
getDiagrams (_, ds, _) = ds

diagram :: Diagram B
diagram = foldr1 mappend $ getDiagrams $ executeTurtleRules rs t
  where rs = genGenerations numGenerations "F"
        t = ((Turtle (0,0) 0), [], Stack [])

By modifying L-System rules, as well as stepDistance, turnAngle and numGenerations constants we can get a variety of different structures.

Fin.

I'm pretty happy with my progress in Haskell, especially with the last mini-project of building functioning L-System.

After two months, I can say that Haskell doesn't feel like a good creative-coding environment for me, strong types look like a great thing for refactoring and maintaining big applications, but when exploring, I felt that it was slowing me down a little. Having to think about type of a thing, when in early stages of exploring the thing is not a very useful thing for me.

I stared less at a blank screen caused by runtime error, and more at compiler output.

Usually, once the compiler was happy, the application was running without issues, so it feels like trading runtime errors for compile-time errors, with the overhead of needing to have more things figured out upfront.

I can imagine using Haskell for building server-side code, or backend for installation, and I hope I'll get relaxed enough timeline on one of my client projects to try it out.

For now, all the code is published online: szymonkaliski/haskell-playground, and I'm moving forward to November's project.

I took 15-45 minutes every day to read through "Learn You a Haskell for Great Good" (free to read online). I was taking notes as I go, and pushed them to online repo: szymonkaliski/haskell-playground

First impressions

Setup was extremely simple, I didn't need anything special besides brew install ghc. vim already supports Haskell, so when compared to Reason (another language I tried out recently), the setup was much easier.

I enjoyed working with ghci REPL similarly to how I like to work with Clojure — in split view with my editor, executing code piece by piece.

"Learn You a Haskell" was a very nice introduction to the language, and as far as I understand it coveres all the basis.

What I enjoyed about the language so far:

1. How terse it is — there's a lot of strange symbols ::, (\, >>=, but syntax itself is quite simple, functions usually are very short, but that doesn't come in hand with them being simple to understand (at least from my newbie perspective)
2. Immutability — I try to write most of my code in a functional and immutable fashion, but having it be enforced by a language, and not a programmer, is a great feature
3. Types — I usually prefer non-typed languages, but Haskell's type system is really interesting, and type annotations are very helpful in trying to understand what functions do; I'm still fuzzy on some details, and plan on doing a bit more reading on type classes

What I didn't enjoy that much:

1. Nomenclature — "applicative", "functor", "monoid", "monad"..., some of the concepts behind the names seem easy, some seem hard, but trying to understand some explanation filled with those cryptic names can lead to a headache
2. There's only a few longer examples in the book, so I haven't actually written that much code yet, but I plan on spending next month exploring some popular libraries (Tidal and Haskell's OpenGL bindings if I get them to work)

Fin.

I doubt if I'll ever get paid to write Haskell, but I'm sure this month wasn't wasted, I learned a few new concepts and looked at things from new perspective which is always a big win.

I'm curious to see how different libraries utilise Haskell type system and how working with bigger pieces of code looks like.

DAS-UI is another node-based experiment (after SDF-UI) that I've built during my one-project-a-month meta-project in 2017.

Intro

I've already built a few node-based UIs and I seem to quite like them (if you're curious about previous ones, you can read the intro to SDF-UI blogpost).

I spend most of my days in CLI, there's something hard to beat about text based interfaces, and I feel I'm much faster with keyboard shortcuts than with pointing and clicking.

Combining both of those worlds has been on my mind for quite some time, but as August began, and as I decided to work on this problem, I had no idea how to start.

Explorations

I've spent a lot of time sketching and thinking, I didn't touch the code for a few days, as I was trying to get at least big-picture view on how this could work.

In hindsight, I was trying to solve to many things at once, I was thinking about UI, UX, and implementation (types? multiple inputs/outputs? streams? user-coded ui? blocks as simple functions? etc...) at the same time. I've finally decided that the biggest exploration here is going to be the UI/UX part, and focused on this for a while.

UI

Since I didn't want to use mouse at all, it became obvious that there needs to be some kind of keyboard-controlled cursor. This cursor has to move by some predictable amounts, so I've added grid. Since there's a grid, I decided to make all the blocks always fit that grid. Blocks can still be dragged around and resized, but they always snap to predefined sizes.

The next important part was tackling the block creation.

I knew from the start that I would have little time to focus on actual blocks implementation, so I decided to allow users to create their own blocks.

So when creating a block, user can select either one from already existing block specs (more on that soon), or create a new spec, with custom functionality, that can later be re-used.

After this was working, the last thing missing was making connections between blocks. Here the biggest inspiration was how vimperator (and similar plugins) handle opening links: you first hit f which adds little tooltips around all the links with few letters, and type those letters to open the link. I used similar mechanism for connecting blocks: you first hover over the input or output, hit c, and then type matching letter to add that link.

Implementation

DAS-UI is another project made with React, just because I know it well, and I know it's not going to block me when implementing new ideas. Whole state of application lives in Immutable.Map:

const initialState = fromJS({
  blockSpecs: {},
  graph: {
    blocks: {},
    connections: {}
  },
  ui: { ... }
});

• blockSpecs is a map, pointing from name, to block spec info (inputs and outputs name), and stringified block implementation
• graph handles actual blocks and connections on board, all of them accessible through uuid
• ui containes all the other UI state, including cursor position, grid size, etc...

There are two intersting parts of the system that I'm going to describe in more details: block specs, and graph engine.

Block Specs

Block specs, meaning the block blueprint of sorts, are simple JavaScript object:

{
  name: 'some/name',
  inputs: [ 'first-input', 'another-input' ],
  outputs: [ 'some-output-values' ]
  code: ({ inputs, outputs, state }) => {},
  ui: ({ state, setState }) => {}
}

• name is an identifier of the block, meaning that each some/name on board will have the same implementation
• inputs and outputs are the arrays of names, they tell the UI to draw relevant items on the screen, and are also used to build the input and outputs objects
• code is the implementation of the block (more below)
• ui is an optional React component that will be rendered inside the block

code is the most interesting part here, this is where the block actually executes some logic. The argument to that functions is an object containing inputs, outputs and state — collections of RxJS streams. These colletions are constructed using inputs and outputs properties — each input and output gets its own stream. In the example above, it would mean that inputs is actually an object:

const inputs = {
  "first-input": new rx.Subject(),
  "another-input": new rx.Subject(),
};

Same thing happens with outputs. State is a stream that connects ui with code. ui can, through setState() call, update the state, which then code is notified about (through state.subscribe). If state was changed from code (with state.onNext), the ui gets re-rendered with updated state values.

This allowed me to keep the whole graph asynchronous, and have most of the piping work being taken care of by RxJS. Block's code can then easily .subscibe() to changes on one of the streams, and push new values out using .onNext(). This might be abuse of how rx.Subject()'s are supposed to work, I'm not an expert here, but it seems to work quite well.

Block specs themeselves are kept inside redux store. This wouldn't be needed if they were constnat, but I wanted users to be able to modify the specs on the go, and easily create new ones. To acheive that, the specs are kept in redux as strings. I've experimented with keeping functions inside the store, and stringifying/parsing them when storing to DB, but that proved to be not reliable and complex in implementation.

Block specs are executed using new Function constructor:

export const executeBlockSpec = (blockSpec) =>
  new Function(`return ${blockSpec.trim()}`)();

This simple function turns the block spec object into something resembling self-invoking anonymous function:

(function () {
  return {
    name: "some/name",
    //...
  };
})();

I can then capture the result, and use the block implementation.

Graph Engine

Graph engine is what actualy runs the calculations. I keep graph as part of redux state: connections and blocks contain everything that exists on the board.

graph.blocks contain block's id, it's position, size, and name of the block spec that it uses. graph.connections contain only reference ids for connected blocks, and names of the input and output.

Graph engine is subscribed to store changes, and listens to changes in the graph subtree. Since the whole state is immutable, I can easily compare if anything has changed (if (!currentGraphState.equals(prevGraphState)) { ... }).

The next important part of the puzzle is immutable-diff package, which generates differences between two immutable structures. I first create the diff, and then walk through each of the changes deciding how to change running graph. The running graph also stores blocks and connections, the difference being that blocks point to actual running implementation (inlcluding reference for input and output streams for piping), and connections point to actual running conneciton, so it can be easily removed (this.connections[id].dispose()). Changes are not applied until the whole diff was walked through, this helped with a few bugs, where blocks would execute code with part-old, part-new state. I collect operations and arguments in futureOps array (for example futureOps.push([this.removeBlock, { id: blockId }])), and then, once I'm done with whole graph, I execute them one-by-one: futureOps.forEach(([op, arg]) => op(arg)).

One interesting thing to implement was a change in existing block spec — I wanted all other blocks using the same spec to be updated. To do that I look for all block ids and connection ids that have anything to do with that block spec, and then I remove them, and re-add them to the graph: services/graph.js

Block's state is also a part of the redux state, this allows me to know when to re-render the block's UI, and I can .subscribe() to changes in graph engine, and pipe them through state stream to block's code implementation.

Fin.

This project, as all previous ones during my one-project-a-month thing, is far from complete.

I think of it as a proof-of-concept that keyboard-based node ui can be usable.

My next idea here would be to move it from web, to Electron, and use filesystem as a source of block's implementations (together with require support, etc..). This would allow me to write low-level block implementation in vim, and then do the high-level piping using something visual. I'd also love to tackle the problem of composing things into sub-patches, and reusing parts of graphs.

If you'd like to hack on DAS-UI it's open sourced and available on my Github: szymmonkaliski/DAS-UI.

DAS-UI is an experimental domain-agnostic keyboard-based visual programming language with JavaScript backend.

It is entirely controlled with keyboard shortcuts, most of which were inspired by vim.

Cursor can be moved with hjkl, new blocks are created with n, connections are made with c. Blocks conform to grid, and can be moved with shift + hjkl, and resized with </> and +/-.

You can read more about process behind making this project on the blog: building DAS-UI.

Code is open sourced: szymonkaliski/DAS-UI.

I spent July 2017 teaching three weeks of different creative technology workshops during summer program at Skyrock Projects in Taipei.

The program was divided into week long workshops: body interactions, no screens allowed and city data. Each week was 2-3 days of heads down — learning technology and programming, and then 1-2 days of student projects. Each of the groups had one week of P5 and Arduino introduction, so they had some basic knowledge of programming.

Body Interactions

The main idea during the first week was to explore interactions that could work with human bodies. We started with exploring some Arduino sensors: accelerometer, pressure sensor, photoresistor and ultrasonic sensor.

Second day was a re-introduction to P5 (running inside of Neutron), and connecting it with Arduino through serial interface. Third day was all about sound (with Tone.js), and coming up with interactions around music and sensors.

I was really surprised by the progress that the students made in just few days, and different final projects that they've came up with.

1. A game where you have to jump over obstacles by physically jumping (using accelerometer) — with an idea to expand this further with pulse sensor to measure the heart rate and promote exercise
2. Stress pong — made using four pressure sensors and LED matrix — that gets harder the harder you press the buttons
3. Memory game with a custom controller, made using a lot of wires and complex logic to help develop your memory

Code: skyrockprojects/Body-Interactions

No Screens Allowed

Second week was all about BareConductive and using conductive paint and capacitive sensors for building interfaces.

We started with exploring Touch Board and playing with some examples.

After that we focused on connecting this to P5 using serial data (and Neutron), to finally connect it to Tone.js.

Final projects included:

1. An interactive map of Skyrock Projects headquarters
2. Music-making T-shirt
3. Touch-based vending machine

Code: skyrockprojects/No-Screens-Allowed

City Data

Last week was a bit shorter, we only got two days of "heads down". During the first day we explored different environmental sensors for Arduino: temperature, humidity, microphone and photoresistors.

At the end of the day we set up data collecting machine that would record the values overnight:

Next day was spent reading back the data, and exploring how we can use it to create images.

Code: skyrockprojects/City-Data

Fin.

I've really enjoyed my month in Taipei, it's a great feeling to see the students progress and explore new ideas.

Workshop structure is especially important here, students can ask questions and experiment when there's no wrong answers.

I also got a chance to test Neutron in the wild — I've added syntax based error checking and better support for Windows, which wouldn't have happened if I didn't seen the problems students had while using it.

It's also nice to take a break from client projects for a while, and take some photos:

Neutron is a self-contained node & npm application made for quick prototyping and teaching.

It borrows ideas from Processing and tries to create similar novice and artist friendly environment.

Intro

Neutron was born out of two main needs:

1. sketching — I love npm ecosystem, almost every piece of code I would want to reuse is there, usually with a few different approaches. This, unfortunately, is not so nice for quick throwaway sketches. I have to set up a project, set up a build pipeline (budo helps here though), open browser, and remember to install dependencies by hand. I wanted to be able to do it quicker, with less steps.

2. teaching — which I really enjoy, I believe that sharing knowledge is important, and that programming is easiest to understand through creative outlets: making graphics and sounds. I always go with Processing since it works everywhere, you just download binary, and run it. At some point, someone always asks about porting the thing they made to web, and even though Java applet is a possibility, usually JavaScript is a better approach — another language they will have to learn, and a new set of libraries to explore; this can be discouraging for newcomers. I wanted Processing environment, but with JavaScript instead of Java.

Tech

Neutron is based on Electron — basically a programmable Chrome window with require() support. This gives us best of both worlds: file system access with require('fs'), full node support, and a window which can render HTML. Neutron wraps this functionality in easy to use application.

On start, Neutron asks to load a JavaScript file, this can be done either by dragging it on the window, or clicking to open file select. This simple UI is build with React, probably overkill in this situation, but this allowed me to quickly move forward.

The most interesting part of this software is an automatic package installation. This consists of two things:

1. finding out which packages are required
2. installing packages with npm

For 1. I used recursive-deps package, after having tested few other solutions. This one was the most straightforward and reliable, and as the name suggests, finds dependencies recursively, which is helpful for sketches spread across multiple files.

Installing packages was trickier, since I knew from the beginning that npm should be a part of the application, and I couldn't just shell out to a globally installed executable.

npm itself doesn't have documented public API right now (it used to have for a while though), so this required some trial and error. Here's the most important gist (install npm as your project dependency first!):

const npm = require("npm");

const packagesToInstall = ["p5", "lodash"];

npm.load(
  {
    color: false,
    progress: true,
    save: true,
    unicode: false,
  },
  (err) => {
    if (err) {
      return console.error(err);
    }

    npm.commands.install(packagesToInstall, (err) => {
      if (err) {
        return console.error(err);
      }

      console.log("installation done");
    });
  }
);

There are some caveats though, from what I've found:

• save: true to save packages in package.json doesn't work, I solve this by creating package.json containing only {} before installing any packages
• there are issues with getting the log info and streaming to the client, I have hacked a semi-solution providing custom node stream that I then send to main process, this works with some issues on OSX, but I couldn't get it to work on Windows
• to avoid re-installing already installed packages on every save, I first use npm.commands.ls to check what's already installed
• I also had to filter out built-in packages from recursive-deps, so for example require('fs') doesn't try to install fs from npm

The full relevant code is here.

Native packages

The biggest problem of this approach is using native packages. If this was based on node, then most packages published to npm are already compiled, and it would be much less problematic. Since Electron uses different V8 version, almost all of native packages have to be compiled after installation. Usually, if node and npm are set up correctly, this is not that big of a deal, at least on OSX (Windows is still a bit more problematic).

The aim of this tool is to allow users to use JavaScript quickly, without any complex setup. I've experimented a lot with bundling some build tools with the app, but sadly, I failed. It seems to be possible on OSX, but I lack Windows-specific knowledge to make this a reality, and I suspect that compilation would still require Visual Studio to be installed.

I've ended up bundling three native packages which I use most often when teaching workshops:

• node-serialport for communicating through serial port
• johnny-five for firmata
• node-midi for MIDI

CI Builds

For building an executable version of the app, I started with local Windows builds on my OSX machine, using Wine. This doesn't work when trying to build native modules — if you need them, you have to compile on yor target platform.

Luckily, this is easily sorted with CI build artifacts! I use AppVeyor for Windows builds, and Travis for OSX ones. As long as the project is opensource, they are free, which is great.

They also work great with electron-builder that I use for building the app. To get this to work, I had to add two config files:

• appveyor.yml for AppVeyor
• .travis.yml for Travis

To have them publish automatically to GitHub, GH_TOKEN has to be set. It can be generated here: https://github.com/settings/tokens, with access right for public_repo. For Travis, it can be added as an environment variable in project settings, and for AppVeyor, it can be included straight into appveyor.yml if it's encrypted: https://ci.appveyor.com/tools/encrypt.

Built-in API

To make Neutron work a bit more like Processing, I've decided to expose a simple wrapper around Electron functions, this allows me not only to get the window size, but move it around the screen, get displays, access mouse position outside of a window, and more.

Detailed API info is up on Github.

Fin.

Building Neutron was painful at times, especially the week I've spent trying to figure out native modules. I believe this tool will be useful for others, for both sketching and teaching.

I'm already using it often enough that I believe it was a good investment of my time. Lately, any idea I have, or anything I want to try, starts with running this bash function:

mksketch() {
  cd $(mktemp -d)    # make random temp directory
  touch index.js     # create empty JS file
  neutron index.js & # I'm using neutron through CLI
  vim index.js       # start playing!
}

Neutron is self-contained node & npm application made for quick prototyping and teaching.

It borrows ideas from Processing and tries to create similar novice and artist friendly environment.

Neutron is based on Electron, so you get everything that Chrome and Node can give you! (Native packages are sadly not supported right now, Neutron ships with ones I use for workshops often: node-serialport, johnny-five and node-midi.)

Detailed info is on github, you can also download it straight from there.

I've also made a longer write-up about ideas and process behind building this: building neutron

Intro

After four months spent working on different tools I wanted a break, and explore a bit more.

ReasonML was on my to-learn list for quite a while, and while a month is not a lot to really learn something, I feel like I at least scratched the surface, and have better understanding and "feel" of this language.

I prefer to learn through doing, and I wanted to explore some generative design ideas I was thinking about. I've spent much more time playing with ReasonML than working on aesthetics, so I hope the presented code might be used as a starting point for doing something more interesting.

Project Setup

I followed original setup guide for getting my toolchain up and running.

Setting up nactual project is very easy, only bs-platform from npm is needed as dependency, and can be started then using local npm binaries: ./node_modules/.bin/bsb. I've set up simple npm script for building and watching for changes, using package.json scripts section:

{
  ...
  "scripts": {
    "build": "bsb",
    "watch": "bsb -w"
  }
}

Another thing that's needed is bsconfig.json in root directory of the project:

{
  "name": "project-name",
  "sources": "src"
}

./src/ folder is source folder where ReasonML code wil be kept, and running npm run build compiles it to ./lib/js/ folder.

This gives us a nice JavaScript code in ./lib/js, but we still need to be able to run it. For small projects I like using budo which also runs browserify so the output code from ReasonML can use npm modules as well!

My final package.json looked like this:

{
  "scripts": {
    "build": "bsb",
    "watch": "bsb -w",
    "serve": "budo ./lib/js/src/app.js --port 3000 --live",
    "start": "concurrently --raw 'npm run watch' 'npm run serve'"
  },
  "devDependencies": {
    "bs-platform": "^1.7.3",
    "budo": "^10.0.3",
    "concurrently": "^3.4.0"
  }
}

This allows me to start serving (and live-reloading) compiled JavaScript using npm run serve, and start both ReasonML compilation and budo with npm start. Bucklescript's compilation (the thing that compiles ReasonML to JavaScript) is extremely fast, and together with budo's live-reload gives near-instantaneous feedback that feels amazing.

It's much faster than typescript or flow compilation, and actually one of the fastest compilers I've ever used.

First project: flow-field

I've started exploring ReasonML through binding to canvas and some other DOM elements. Unfortunately, the official docs are lacking here, I tried to make sense of Bucklescript docs but they are written for Ocaml, not Reason, so that was tricky at first. What helped a lot was browsing through existing ReasonML projects:

• bs-express bindings for express
• reactRe bindings for React

I ended up writing just the bindings I've needed, first for Document:

let module Document = {
  type element;

  let window: element = [%bs.raw "window"];

  external createElement : string => element = "document.createElement" [@@bs.val];
  external appendChild : element => element = "document.body.appendChild" [@@bs.val];

  external addEventListener :
    element =>
    string =>
    (unit => unit) =>
    unit = "addEventListener" [@@bs.send];

  external getWidth : element => int = "innerWidth" [@@bs.get];
  external getHeight : element => int = "innerHeight" [@@bs.get];

  external setWidth : element => int => unit = "width" [@@bs.set];
  external setHeight : element => int => unit = "height" [@@bs.set];

  external requestAnimationFrame : (unit => unit) => unit = "requestAnimationFrame" [@@bs.val];
};

Few notes:

• unit is "nothing", and (unit => unit) means function from nothing to nothing, this is used for callback functions
• [@@bs...] are Bucklescript annotations for FFI

This allowed me to create canvas and set its size:

let canvas = Document.createElement "canvas";

Document.appendChild canvas;

let setCanvasSize () => {
  let width = (Document.getWidth Document.window);
  let height = (Document.getHeight Document.window);

  Document.setWidth canvas width;
  Document.setHeight canvas height;
};

Document.addEventListener Document.window "resize" setCanvasSize;
Document.addEventListener Document.window "DOMContentLoaded" setCanvasSize;

I've made similar simple bindings for canvas: src/Canvas.re. Since every file is a module in Reason, we can drop let module Canvas = if saving code in separate file.

Flow-field app logic was quite simple to write, the basic algorithm is:

1. create bunch of points (notice nice Random.float functions from Reason core)
2. update point's velocity based on flow field direction in its current x/y position

For flow-field direction, I use Simplex noise algorithm which generates one value based on two inputs, similarly to Perlin noise. I calculate vector from this value, treating it as angle:

let angleToVec angle : vecT => {
  x: (sin (angle *. 2. *. Math.pi)),
  y: (cos (angle *. 2. *. Math.pi))
};

Notice the vecT annotation, this allows compiler to be sure that returned value is { x: float, y: float }.

I've spent most time building this example exploring ReasonML syntax, and fighting the compiler.

Coming from working mostly in non-typed languages (I've been using JavaScript and Clojure lately), it felt like doing more work for little to no profit.

This was noticeably slowing me down, to the point that I probably won't ever use ReasonML for experimental/exploratory projects. At the same time, I'm sure that type safety would be great for any bigger UI application.

Second project: metaballs with regl

I've added a simple collision detection to the first project and got laplacian growth out of it:

Basic algorithm is:

1. start with a fixed particle, and a lot of "floating" particles
2. if floating particle hits the fixed particle, it becomes fixed as well
3. repeat until you run out of particles

I got curious how this would look if the particles were metaballs — I was looking for more organic "blobby" visuals.

To create metaballs in 2d, we have to calculate distance to all the particles for each pixel, this would perform very badly in CPU-land, so I've decided to switch to webgl. Regl was another cool project on my to-learn list, so I decided to combine the two — learn about regl API while binding it to ReasonML, and then use it for metaballs.

I've spent a lot of time on writing a nice, usable wrapper. Regl is great to use in JavaScript, but I had a lot of problems with forcing Bucklescript and ReasonML types to cooperate. Final wrapper is very hacky, and I'm both ashamed and proud of it, some interesting parts:

let create : Document.element => unit = [%bs.raw {|
  function(canvas) {
    // first hacky thing, expose regl as top-level window things
    window.regl = require('regl')(canvas);
  }
|}];

This exposes regl as top-level window thing, so I can then use my wrapper as a singleton. Not sure if that's good or bad idea, but has worked for me.

external makeDrawConfig :
  frag::string? =>
  vert::string? =>
  uniforms::Js.t{..}? =>
  attributes::Js.t{..}? =>
  count::int? =>
  drawConfigT = "" [@@bs.obj];

let makeDrawCommand : drawConfigT => drawComandT = [%bs.raw {|
  function(opts) {
    var command = window.regl(opts);

    /* expose ".draw" so we can use draw external */
    command.draw = command;

    return command;
  }
|}];

external draw : command::drawComandT => uniforms::Js.t 'a? => unit = "" [@@bs.send];

This allowed me to create JavaScript object using ReasonML syntax:

let drawTriangleConfig = Regl.makeDrawConfig
  frag::triangleFrag
  vert::triangleVert
  count:4
  ...

Then use this configuration object to create hacked draw command:

let drawTriangleCommand = Regl.makeDrawCommand drawTriangleConfig;

Which finally can be used in Regl.frame to draw:

let draw t => {
  Regl.clear
    color::[| 0, 0, 0, 1 |]
    depth::1;

  let triangleUniforms = [%bs.obj {
    width: state.window.width,
    height: state.window.height,
  }];

  Regl.draw
    command::drawTriangleComand
    uniforms::triangleUniforms;
};

Regl.frame draw;

Full wrapper is here: src/Regl.re, and final results below:

Fin.

I enjoyed this month's exploration and learning a new language as nice break from tool-making research.

ReasonML is an interesting project, I love how fast the compiler is, how readable the output code is, and how types could help while working with data structures and complex applications.

It still feels like early days though, with not the greatest documentation, and only few example projects/libraries using it. Right now, I would consider using it for UI frontend work, and I'll keep my eye on its future developments.

If someone is interested in making the regl wrapper better — feel free to contact me. I'd love to help, but I probably won't be pursuing this myself any time soon.

Timav is a tool for analysing time tracking data collected in Google Calendar.

It evolved organically to fit the work I do, and how I do it, and should probably be used as reference to building your own system.

You can find more details in this blog post: time tracking.

Code is open sourced: szymonkaliski/Timav.

It evolved organically to fit the work I do, and how I do it, and should probably be used as a reference to build your own system.

Intro

I've built my first time-tracking system when I finished college and jumped into freelance work. I had two main reasons to build it:

1. I wasn't sure how much things should cost, and I hoped that over time, by analysing which projects I felt satisfied with, and knowing how much time I've spent on them, I would know how much a day of work is worth. After over four years of doing this, I know that this is not as simple as it seems, but it was a start.

2. I wasn't sure how much time things can take, and I knew that potential clients would be asking about timelines. I figured that the more projects I make, and the more data I have, the better I can approximate things better. Again, I know it's not that easy, but it was something to start with.

I was also curious how much time I would spend working daily, and how it changes over time.

Since then I've been rebuilding this system yearly, as I've realised how much impact on me it has — "We shape our tools and then our tools shape us.". My ideas about what I want to track keep changing as well.

Timav is actually a fourth version in this process, and I'm pretty sure it's not the last one.

I'll start with some history of previous versions, and then get into more details on how I approach time tracking now.

First attempt

Because my first time tracking system has been built before I've started tracking time, I have no idea how long it took to build it. The idea was pretty simple though, I wanted to store all the logs in a database, and be able to query and analyse them. The log consisted of starting and ending times, project's name that it belonged to and type — either personal or work, later another type, research was added, but I didn't use it much. I also tracked payments with date, amount and project name. Everything was held together by project name, so my tracking was only concerned about how I work on very specific things.

During the day, I would jot down when I started and finished some task. I've tried to be as honest with myself as possible, so whenever I've started browsing Hacker News or Reddit, I would have noted down that time as end of log. Usually after a day of work I've had something like this:

Project work 10:15-10:45
Project work 11:00-11:30
Project work 12:00-13:15
...

I would then go to the browser, and input one-by-one all this data:

I've had similar UI for inputing payments, and boring but informative table with info about all my projects:

I've also tried to get simple stats out of the data, but haven't used it much:

This doesn't look terribly exciting, but had worked well for me for over two years. There was one big problem in how this system was built, mainly this table of projects. At some point I changed it to be sorted by how much I made per hour, so all my personal projects, which earn me nothing, would be at the bottom of the list.

After two years of using it, I stopped working on any of my own things, and for a while couldn't figure out why. I got a bit depressed, and I couldn't force myself to do anything other than work. It turns out that working on personal projects is very important for me, and seems to be connected to the quality of my life.

The "eureka" moment came after watching Devine Lu Linvega's talk Frameworks for Mystics, and hearing how he uses his time tracking to split time between working with audio, visuals and programming. It was one of strangest mixes of emotions I had, understanding that the tool I made for myself, that should help me, ended up impacting my life in such a big and unpredicted way.

My solution to this problem was simple — to build a new tool, one that would show me things from different perspective.

Second attempt

Unfortunately, timeline on this is very messy, second version of this tool was developed somewhere during second year of using the first version. I planned on re-writing it in React (old version was in Backbone), playing with how data is displayed and adding more charts. I was also thinking of some kind of external API so I could add logs from command line. I only spent around 20 hours on this, and it died together with few other projects during that no-personal-projects time.

Third attempt

So, back to the main story.

Third attempt started just after understanding what the old version was doing to me. Distinction between personal projects and work was an important thing to tackle, I also wanted to have more open-ended system, allowing me to track more things, and adapt to what I'm doing at the moment.

The friction of noting down and inputing events into web-based UI was also getting in my way, so I decided to use calendar for logging — idea stolen from Marcin Ignac.

The system for describing logs looks like this:

[Project] @tag1 @tag2(subtag)

• Project is optional, and allows me to group logs for given project, it's usually a client project I work on, or well-defined personal project
• @tag is additional metadata that I can use to analyse the logs, I can add how many tags I want to a given event, but it's usually just a few; at the moment most often used ones are: @work, @personal, @language() (I track what programming languages I use)
• @tag(subtag) is a shorthand for having more data about a given tag, but allows me to still nest them under one bigger thing, this might be overly complex, but works for me — I use @language(js) when I'm writing JavaScript, @language(clj) for Clojure, @health(gym) for when I'm training and @health(walk) when strolling

In Timav I also added full-day logs that describe events. Event can be a big milestone or deadline (I'm adding event notes to describe what it was about) or a payment (I use special tag @cash(), so @cash(1000) would mean I earned 1000PLN on a given day).

Third version was also a big technology change, I used Clojure and Clojurescript together with Neo4j, just because I wanted to learn them. It took around 80 hours to build, and was used for over a year. The biggest problem was performance — it was my first big thing built with Clojurescript, and I didn't pay much attention to how well it worked. Turns out that if I have to wait for a minute for the app to load, I'm not going to use it that much.

But it a had few interesting ideas, first of all I had a chart that would show me how I've spent my time between most important tags for me at the moment (here @work and @personal):

I've also built this chain view which showed me each day of the year, with red representing @work and blue @personal:

You can clearly see the 2015 when I wasn't doing any personal projects, and how building third version at beginning of 2016 pushed me to work for at least few minutes on some personal project every day.

The idea has worked — by changing the tool, I changed what I do.

The performance was still a problem, and since I started doing one-project-a-month this year I wanted to finally improve, and open source it. That's how Timav was born.

Timav

Timav is built around ideas that have been collected for last four years, it's still work in progress, and I plan on getting back to it later this year. For now it provides two views:

1. Projects — with info about logs connected to the projects, featuring a simple chart of time spent, and cash calculations if there are any
2. Chains — which allows me to track last couple of days with given tag, this lets me build up momentum and create new habits

All logs still come from Google Calendar, so on the first run you'll have to authorise application access in Settings, and choose calendar where tracking data is stored.

Fin.

Timav took 29 hours to build, but is far from complete.

I'm not sure about the Chains UI, I would love more detailed info about the tags in Projects view, and I've been thinking of explorable Gantt Chart for all the projects, so I can see when I'm overworked. I'm also missing the big-picture view from version three, where I could see the progress since the beginning.

If you want to hack on Timav, it's open sourced and available on my Github: szymonkaliski/Timav, but honestly, it's probably better if you build your own, I wouldn't want my ideas to limit yours.

Intro

When I work with creative generative systems, it feels like I spend as much time fiddling with different knobs as I spend writing code.

By knobs I mean not only GUI sliders, but also code constants, array sizes, etc. The bigger the system is, the more knobs it has, and it becomes harder and harder to explore. It also usually feels like the nicest results come out of extremely specific set of values, and once I find them, it's scary to start fiddling with them again.

I always wanted to able to move through this parameter space fluidly and see the results in real time. To collaborate with the machine, instead of setting everything by hand.

In February I explored evolutionary algorithms for generating simple GLSL code in WallGen, and I started wondering how much of that evolutionary process could be abstracted and re-used for more general purpose.

AST

First step in making this a reality was to detect all the numbers in code. I could have used regular expressions, but it's usually a bad idea when it comes to very complex structures (like code).

A proper way of analysing most of programming languages, is parsing them into abstract syntax tree - data structure describing the source code.

This tree can be then traversed and transformed, and as long as it's kept in proper shape, it can be pretty-printed back into text form. So a simple identity function on code would turn it into AST and back into code: code -> AST -> code.

I've spent a few days trying out different libraries for building and transforming ASTs, and settled on recast. Parsing with recast is pretty simple in it's most basic form:

const ast = recast.parse(sourceCode);

recast.visit(ast, {
  // visit Literal - numbers, strings, etc...
  visitLiteral: function (node) {
    // node.value has raw and parsed value,
    // node.value.value is parsed: 17 instead of "17"
    if (isNumber(node.value.value)) {
      console.log(node, "is a number!");

      // I now know that node is a number, so I can modify it
      node.value.value = Math.random() * node.value.value;
    }

    // tells recast to keep on going
    this.traverse(node);
  },
});

I tested this on few simple P5.js sketches, it worked well until I hit loops, and colors.

As for the loops, the problem is that numbers can be used to iterate over array, and array[1] exists, but array[0.3551355] does not:

for (var i = 0; i < 10; i++) {
  points[i].draw();
}

If starting or ending conditions get changed to floating point numbers (or numbers outside of points.length) this would break horribly, with the infamous undefined is not a function. I had to find a way to detect if number is part of for/while loop, there are two easy ways in which this could happen: for (var i = 0 ..., and for (i = 0 .... It's enough to walk few times up in the tree, and find out whether parent node is for or while.

Unfortunately there's one much more problematic case:

var n = 10;

...

for (var i = 0; i < n; i++) { ... }

I've spent a good few days on this one, and I'm still not sure whether my approach is the best one, but it seems to work. Whenever AST finds a number, which is a part of variable declaration (var i = 10), I ask AST for a whole scope in that place, and traverse it, looking for anything that involves this variable's name (i in this example). When I find something, I test again if it's a part of for/while, and act on that number differently.

Relevant piece of code can be found on github: code-transform/index.js. If you know a better way, please submit an issue or pull request.

Colors are a simpler version of this problem, for example turning background(100) into background(300) (or worse background(-100)) doesn't make any sense. Numbers found inside of few specific functions (fill, stroke, background, etc...) should be limited between 0 and 255.

So, another check that I run finds out if the callee (function that calls some piece of code) matches one of pre-defined functions. We have two basic cases here, simple one: fill(10, 20, 100), and complex one, same as with for/while loops:

var r = 120;

...

fill(r, 10, 20);

I tackled it exactly the same as with loops, but instead of looking for for/while I look for callee names.

A relevant piece of code can be found on github: code-transform/index.js.

Both solutions are not ideal, but cover most of the cases I could find.

Final code analysis and transformation has to be split into two parts to fit the evolutionary algorithm approach. I find numbers and their types first and run evolution on this list (with mutation taking in account the type of a number). When new set of numbers is evolved, I generate a new code, using the modified array and traversing through the input again, this time replacing numbers at matching indexes.

Displaying the results

Another thing to tackle was how to show more than one sketch at a time. The simplest solution would be to force users to use instance mode and provide different containers for each of the sketches. My aim was to make the tool as frictionless as possible and to not force users to change anything about their code, so I thought of using <iframes> to contain the sketches. I wanted to have this running without a backend, so responding with different HTML documents containing different modified sketches for different URLs, would be tricky to do. Luckily there's srcdoc attribute on <iframe> in html5, and the final rendering code became very simple:

const generateHTML = (code) => `
  <style>
    * { margin: 0; padding: 0; }
  </style>

  <script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/p5.js/0.5.7/p5.js"></script>

  <script type="text/javascript">
    ${code}
  </script>
`;

const Renderer = ({ width, height, code }) => (
  <div>
    <iframe
      scrolling="no"
      width={width}
      height={height}
      srcDoc={generateHTML(code)}
    />
  </div>
);

Attempt at automated testing

While I was working with AST part of the project, I wanted to test on as many different P5.js sketches as possible. Someone on twitter suggested using openprocessing as a source of sketches, and I quickly wrote this simple scraper: openprocessing-scraper. There's over 500 megabytes of sketches there, with over 2700 P5.js ones, it didn't make sense to test them all by hand.

I experimented with simple electron application that would load all the sketches in directory one-by-one and report on errors. Unfortunately, it turned out that most of the sketches reported as broken had already been broken before my AST transformer did anything. On the other hand, errors that I was looking to find wouldn't necessary show up as JavaScript errors, so I gave up on this for now.

In case anyone would like to take this idea further, I left it in the repo: code-transform-electron-tests.

Fin.

This project was the first one this year that really pushed me to learn something new — traversing and transforming JavaScript ASTs.

I've spent more time exploring and failing, since I knew all other parts were either already there (evolution), or easy to make (UI).

As usual, there's a lot that could be improved, and I think with relatively little work, this could support different creative-coding libraries as well.

You can play with live version here: http://szymonkaliski.github.io/parametrium.

Code is open sourced and available with MIT license here: szymonkaliski/parametrium.

Parametrium is a parameter space explorer for P5.js sketches.

It allows users to interactively move through parameter space of all the code constants, without modifying anything in code.

You can find more details (including info on AST transformations) in this blog post: building parametrium.

Live version is running on github pages: http://szymonkaliski.github.io/parametrium, and code is open sourced: szymonkaliski/parametrium.

Intro

My favourite desktop wallpapers are calming blurry colors, but it's hard to find them and I lack enough Photoshop skills to make something I enjoy looking at.

My solution to this "problem" was to try to generate them with code, ideally in a way that I could potentially get an endless stream of nice soothing abstract blurs.

Generating blur

The first step was to actually get something on a screen, and to get nice and performant fullscreen gradients, I jumped straight to GLSL. It would be possible to use CSS gradients for example, but for anything more complex than linear/radial gradient it would get tough.

I thought of these kinds of blurs as a set of points which emanate colours. Combining multiple points with nice color palette and proper placing together could create images similar to what I was looking for.

I got to what seemed like a good approach using distance and mix in GLSL. Most basic gradient looked like this:

Rendered with this small piece of code:

precision mediump float;

uniform vec2 iResolution;

void main() {
  vec3 color1 = vec3(0.75, 0.31, 1.0);
  vec3 color2 = vec3(0.232, 0.42, 0.529);

  vec2 gradientPoint = vec2(0.23, 0.47);

  vec2 pos   = gl_FragCoord.xy / iResolution.xy;
  float dist = distance(pos, gradientPoint);

  gl_FragColor = vec4(mix(color1, color2, dist), 1.0);
}

Generating a lot of blurs

Getting one shader to work was the easy part, but I wanted endless, similar, but not exactly the same wallpapers.

I started toying with this idea few months back, first approaching this as a k-NN problem, with a logic resembling this one:

1. generate a lot of random wallpapers
2. allow user to select a set of "target wallpapers"
3. keep coming up with random wallpapers until they are close enough (k-NN) to the set of target wallpapers

This has worked awfully — basically, I had been trying to keep coming up with random values until I found a subset close enough to what I like.

The next step was to try to reverse this process — to only come up with random values near the "target" set. An evolutionary algorithm seemed like an obvious solution — starting with random population of wallpapers and steering the process toward user-selected goal.

Daniel Shiffman has a great explanation of genetic algorithms in his "Nature of Code" book: natureofcode.com/book/chapter-9-the-evolution-of-code/ so I'm not going to go into the details here.

The most challenging problem in interactive evolution is coming up with a nice balance between number of choices presented to the user and calculations based around fitness function. I didn't want to ask user to rate each of possibly thousands of generated wallpapers, and I came up with a simple solution — the user selects just one wallpaper to "evolve", and fitness of the rest of the wallpapers is calculated as distance from user selected one. Wallpaper DNA is represented as an array of normalized floating point values, so calculating distances between these arrays is easy, although far away from what happens in nature — I'm not even comparing phenotypes, and two arrays which are very similar in DNA could potentially yield very unalike images. But it was good enough for this experiment, and I couldn't come up with any other ideas anyways.

Make it fast

I try to follow simple set of rules when coding: make it work, make it right, make it fast.

After getting to the point where my generated images started to look similar but different to the previous choices, and after making the code readable and functional (everything operating on immutable arrays and returning new immutable arrays) I ended up in a situation where bigger populations could evolve for few seconds, and since JavaScript is single-threaded it would block the browser. I had been trying to optimize the code for an afternoon, but couldn't speed it up by a factor of 10x, so I decided to move the calculations to webworker — basically a separate thread to execute JavaScript in, which can communicate asynchronously with the main rendering thread.

Since I used Immutable.js for handling the data, I would have to first convert them to plain JavaScript values, then into JSON and do the opposite in webworker, and then after calculations are done do it again in another direction. Stringifying with JSON can get quite slow, and since I could potentially use thousands of arrays, each with hundreds of floating point values, I was getting worried of potential bottleneck. Luckily I stumbled upon transit-immutable-js which can turn immutable structures into transit strings directly.

Still, after doing some tests, running evolution in webworker is slower than running the same code on a main thread (consistently by few percent). I suspect it's because of an overhead of transit-ing and transmitting data, but it gives us an illusion of things happening much smoother, which is worth few extra milliseconds.

Make it pretty

After each evolution step, the whole population is replaced with a new one. I wanted to make this easy on the eyes, and since each genotype is an array of numbers (with constant length) it was easy to "morph" them into each other.

I render GLSL with amazing gl-react so I just had to re-calculate that genotype array to get my transitions. I wanted to make this automatic as well, and I ended up with a simple loop function powered by requestAnimationFrame (I'm using raf here) — every time the phenotype component receives new genotype data it starts the update loop, which goes on until current rendered values equal the target ones. Thanks to Immutable.js comparing two arrays for equality requires just .is(), and is very fast.

Below is a relevant part of phenotype component class, whole file is on github: phenotype.js.

updateLoop() {
  const eps = 0.01;
  const k   = 0.9;

  // move towards target code
  const code = this.state.code.map((v, i) => {
    const dist = abs(v - this.props.code.get(i));

    return dist < eps ?  this.props.code.get(i) : k * v + (1 - k) * this.props.code.get(i);
  });

  this.setState({ code });

  if (!code.is(this.props.code)) {
    // keep updating if current code is different from target one
    this.rafHandle = raf(this.updateLoop);
  }
  else {
    raf.cancel(this.rafHandle);
    this.rafHandle = undefined;
  }
}

componentWillReceiveProps(nextProps) {
  if (!nextProps.code.is(this.state.code) && !this.rafHandle) {
    // start animating if target differs from current code
    this.rafHandle = raf(this.updateLoop);
  }
}

Fin.

This project was simpler than January's SDF-UI, and took 6 hours less to make (totaling in almost 24 hours of development time). There was a lot to figure out, and I ended up learning about webworkers, and implementing my first evolutionary algorithm after I finished college.

Like the last time, there's a lot that could be improved, but March is almost there, and I'm going to move on to something else.

You can play with live version here: http://szymonkaliski.github.io/wallgen.

Code is open sourced and available with MIT license here: szymonkaliski/wallgen.

Wallgen is an evolutionary wallpaper generator, using genetic algorithm to create never ending list of abstract ambient wallpapers.

Selecting one of initial random wallpapers to evolve makes it the most fit one in the current population, giving it the biggest chance to crossover (exchange genetic material with another wallpaper). Wallpapers also mutate with small random chance, and can be easily downloaded.

You can find more details in this blog post: building wallgen.

Live version is running on github pages: http://szymonkaliski.github.io/wallgen, and code is open sourced: szymonkaliski/wallgen.

(Pre-) Intro

I have a quite common problem of not finishing personal projects. I tend to start a lot of them, and once the main problem is semi-solved (or I get bored), I move on to something else, leaving no trace of having anything done. I usually learn something in the process, but it would be much nicer, and hopefully more useful, to have something to share.

In 2017 I decided to work on a single personal project every month, with strong incentive to publishing it at the end of that month, and SDF-UI is first project of that kind.

Intro

You can read more about SDFs and why they are cool on the project page and all over the internet, so I'm going to skip that part.

I was experimenting with this technique for generating images in November last year, and one of the biggest problems for me was that after a bit of fun, modeling function would turn into a mess similar to that:

return vec2(
    max(
      -sdBox(p, vec3(r + .01, r * 2. - c, r - c)),
      max(
        -sdBox(p, vec3(r - c, r * 2. + .01, r - c)),
        max(
          -sdBox(p, vec3(r - c, r * 2. - c, r + .01)),
          max(
            -sdSin(p - vec3(0., r - 1.0, 0.)),
            sdBox(p, vec3(r, r * 2., r))
            ),
          )
        )
      ),
    0.0);

Obviously, this could have been written much nicer, but then all the parts of the scene must have been named (box1, box2, etc...), and this additional thing to keep in mind doesn't compose well with play mindset, at least for me.

Second reason why I wanted to work on this is that I've already built few node-based UIs, one for building interactive movies, and then two much more interesting ones. First one is MIDI preprocessor for HLO which allows users to modify MIDI messages on the fly and route them between hardware (MIDI keyboards, Arduinos, etc..) and software (OSC, virtual MIDI).

Second project is SAM 2.0 for SAM Labs - a visual programming language for connecting different bluetooth modules together, aimed at kids. Unfortunately both of these projects aren't finished yet, and it doesn't seem they will see the light any time soon.

Graph

When working on projects like that, I like to start with the thing that I know I have no idea how to do.

In this case that was turning the graph structure into linear shader code.

I started with sketching ideas for graph structure:

• nodes represent GLSL functions (sdSphere(), opRepeat(), etc.)
• nodes can have multiple inputs (function arguments), but only single output (float, vec2, vec3, etc.)
• node output can be connected to multiple inputs, but node input can only have one incoming edge (if you imagine input as function argument, there's no easy way of putting two values there at the same time)

Basic JSON representing this structure looked like that:

{
  "nodes": {
    "n001": { "id": "n001", "type": "sdSphere" },
    "n002": { "id": "n002", "type": "sysInput" },
    "n003": { "id": "n003", "type": "sysOutput" }
  },

  "edges": {
    "e001": {
      "id": "e001",
      "from": { "id": "n001" },
      "to": { "id": "n003", "inlet": "d" }
    },
    "e002": {
      "id": "e002",
      "from": { "id": "INPUT" },
      "to": { "id": "n003", "inlet": "p" }
    }
  }
}

This would represent simple connection: sysInput -> sdSphere -> sysOutput.

I kept this JSON in a file, and started working on ideas how to unwind the graph. I ended up with solution where I start from sysOutput node (I know this is the output of the system), and build up by finding what is connected to its input, then by what is connected to that node inputs, and so on. This recursive function is in compile-graph file.

Walking the graph backwards is unfortunately not enough to build the shader code — we need to store and access GLSL code that given node type represents: sdSphere has to be turned into sdSphere(p, pos, r) etc.

Second part of the system is a function which generates class for node from simple spec file. All nodes are created like that:

export default generateNode({
  spec: {
    inlets: [
      { id: "p", type: "vec3", value: "vec3(0.0)" },
      { id: "r", type: "float", value: "0.5" },
    ],
    outlet: { id: "d", type: "float" },
  },

  frag: `
float sdSphere(vec3 p, float s) {
  return length(p) - s;
}`,

  generate: ({ p, r }) => {
    return `sdSphere(${p}, ${r})`;
  },
});

• spec part describes inlets and outlet of that node
• frag is part of fragment shader that will be injected into final GLSL code
• generate is function which receives all inlet values and returns string that will be injected into modeling part of the GLSL code

You can have a look at internals of generateNode. I think most interesting part is little trick with overwriting toString of returned class. This allows me to simply plug instances of different classes into each other exactly as if I was concatenating strings.

For example, in sdSphere, if r is connected to mathSin which is connected to sysTime, it will first run toString() on mathSin, which in its generate function will run toString() on sysTime which will return string time (uniform, but that's not important), thus turning this whole chain into sdSphere(p, sin(time)). On the other hand if r isn't connected to anything it will grab default value for this input — string 0.5 turning final code into sdSphere(p, 0.5). Neat, right?

One thing I wish I could spend more time on was designing the actual DSL that I've created. I tried to keep an eye on things happening organically — I ended up only using float and vec3 types, and tried to keep naming and conventions consistent — nodes generating surfaces always have point p as first input, distance is always named d, etc.

UI

Once GLSL generation was in place, I started working on the UI.

One month seems like a long time, but I only get 1-2h a day tops for working on my own things, so I had to be pragmatic if I wanted to finish this in time.

I chose stack that I know well: react, react-gl, redux and immutable.

I augmented initial JSON with x and y positions of each node, and started drawing things on screen — first nodes in its proper positions, then connections between them as SVG lines. Dragging is performed by dynamically adding mosemove event on mousedown, and removing on mouseup — something I've found to be most reliable. You can look at onMouseDown in editor-node file.

One of tricks I came up with, inspired mostly by om, is to keep almost everything in the redux store. This gives us nice property of being able to dump it in localStorage on every change, and load it from there every time the app is loaded, which gives us something resembling cheap code hot-replace. It's still easier to set up and manage than HMR in my experience.

// check if we are in "debug" mode (http://localhost:3000/?debug)
const isDebug = window.location.search.indexOf("debug") >= 0;

let parsed;

if (isDebug) {
  try {
    parsed = JSON.parse(localStorage.getItem("state"));
  } catch (e) {
    console.error(e);
  }

  // helpful global function for clearing the state
  window.clearState = () => {
    localStorage.setItem("state", null);
    window.location.reload();
  };
}

const initialState = parsed || defaultState;

This helpful thing allowed me to play with design and debug without having to re-create the state every time from scratch.

Second nice redux trick is keeping all actions in a hash-map, it's easy then to find typos and unhandled actions:

const actions = {
  ADD_NODE: addNode,
  MOVE_NODE: moveNode,
  DELETE_NODE: deleteNode,
};

export default (state = initialState, action) => {
  if (actions[action.type]) {
    state = actions[action.type](state, action);
  } else {
    console.warn(`No handler for action ${action.type}`);
  }

  return state;
};

Fin.

I've finished this project in exactly 30 hours of coding, which I think is pretty good for what it can currently do. It's a nice feeling when things start clicking together, and at the end I was able to add bunch of different nodes quickly, because the architecture has been prepared for that since the beginning.

There's obviously a lot to improve on, but this has never been intended to be a finished product, but more of an inspiration and sample code for people wanting to experiment with building their own node-based DSLs.

You can play with live version here: https://sdf-ui.firebaseapp.com/, it's good idea to open HELP if you are visiting for the first time.

Code is open sourced and available with MIT license here: szymonkaliski/SDF-UI.

SDF-UI is a node-based DSL for generating complex shapes using SDF, GLSL and WebGL.

SDF stands for Signed Distance Function - a function which returns the shortest distance between given input point and some surface. Sign of returned value indicates if point is inside, or outside that surface. Complex scenes can be defined in terms of SDFs and calculated using raymarching.

One of the nicest properties of SDFs is ease of manipulating shapes with constructive solid geometry (CSG): intersection, union and difference can be represented as simple operations on distances from two surfaces: example of CSG operations with SDF-UI

For more explanation on SDFs visit:

• https://en.wikipedia.org/wiki/Signed_distance_function
• http://jamie-wong.com/2016/07/15/ray-marching-signed-distance-functions/
• http://iquilezles.org/www/articles/raymarchingdf/raymarchingdf.htm

Project was made with react, react-gl, redux and immutable. Some of the SDF/GLSL code was borrowed from stack.gl and hg_sdf.

Live version is running on firebase: https://sdf-ui.firebaseapp.com/, and code is open sourced: szymonkaliski/SDF-UI.

You can read more about process behind making this project on the blog: building SDF-UI.

I've spent last two months of 2016 teaching intensive eight week creative coding course at OFCourse in Shanghai.

All my previous experiences with teaching were short workshops (2-3 days), so preparing enough material for eight weeks was an interesting challenge. Classes included Processing, Arduino and Creative Lab, where students worked on their own projects.

It was exciting to go deeply into topics, explore different aesthetics and interactions, something that's not possible during shorter workshops.

On Processing side, we started with introduction to programming, explored functions, and later classes. Once we got good foundation for how objects work, we used Minim for making sounds, OpenCV for interacting with camera, and even some Machine Learning with Wekinator.

Particles interacting with mouse made as introduction to arrays

Introduction to edge detection with OpenCV

Live camera image classification with Wekinator

We explored world of electronics with Arduino, starting from simple LED examples, through segment displays, LED matrixes, to working with different types of motors, and most importantly communicating with Processing through serial connection. This last part was very inspiring for students, and almost all of final projects featured some kind of interaction with physical world.

One of midterm projects made by students

From my experiences, the hardest thing for students to understand was scope, where variables come to life, differences between global and local, and everything connected to that.

It also seems very strange that even with all the tools we have, it's still near impossible to teach without markers and whiteboard.

I'm glad I had the opportunity to be involved in this course, I'm a firm believer in giving back to community and sharing what I know. It's also very encouraging to see students making progress, and having those little eureka moments when things start to click.

Code we created during workshops is available on github: ofcourseio/OFCourseFall2016

I collaborated with Bare Conductive in building software for Pi Cap which brings capacitive sensing known from their TouchBoard to Raspbbery Pi.

Pi Cap has 12 electrodes for sensing capacitive touch, high quality audio output, prototyping area for soldering, button, and RGB led.

Our main task was to prepare software examples for users to quickly start playing with this board in C++ or Python. We also created Python wrapper for C++ MPR121 driver prepared by Stefan Dzisiewski-Smith from Bare Conductive.

Examples range from controlling RGB led and reading input from the button, to more complicated things, like streaming data through OSC, playing polyphonic sounds, and even text-to-speech.

All examples are open sourced and available at Bare Conductive's github account.

I created unique hardware, and custom software, which let Piotr Bejnar play a concert on human heartbeats.

We built 12 bracelets reading pulse and movement data in real-time, custom software which controlled tempo of Piotr's music, and another application for live generative visualisations.

Bracelets were powered by Arduino Pro Mini with pulse sensor, accelerometer, and 868MHz radio. They communicated between each other and "main" radio transceiver via meshing network - even if one of the bracelets didn't "see" the master radio, it could communicate with it using other nodes.

Each bracelet was equipped with four RGB LEDs changing color based on a bracelet status - blue indicated that bracelet is working, and red meant connection to the DJ - the tempo from that bracelet is controlling tempo of a song.

Casing was 3D printed, and as straps, we used Apple Watch ones.

We wrote a lot of software for this project, it could be divided in main three parts:

• main transceiver - handling communication between all three computers and all bracelets through radio transceiver
• audio controller - receiving data from main transceiver and controlling Ableton Live tempo
• generative visuals - receiving the same data from transceiver and generating visuals on the fly

Communication between all the machines was handled with OSC through ad-hoc network.

An audio controller, created in max4live, was running on Piotr’s machine. It allowed him to easily choose which bracelet currently controls tempo, and showed all data in real-time. While playing, Piotr was switching between active bracelets which were in turn controlling the song tempo.

Live visualas were important part of the whole project, they intended to show people that the actual tempo data is being controlled by the bracelets. We closely collaborated with amazing VJ and graphic/motion designer - Michał Mierzwa - to build half-generative visuals.

Part of them was made with fully geneative code in clojure and quil (processing wrapper), and another part were Michał's video samples created specially for that occasion. Live visuals application was rendering frames through Syphon, which allowed Michał to receive and display them easily as a part of his Modul8 workflow. In addition, Modul8 tempo was also synced to current bracelet tempo, making the whole concert synced to live heartbeat data.

glsl-auto-ui is an experiment in automatically generating UI components for DAT.GUI using GLSL uniforms.

It's an attempt at drastically shortening iteration time with (almost) immediate feedback.

The code is open sourced: szymonkaliski/glsl-auto-ui

I collaborated with Variable studio and Squint Opera to build online interactive application showing the next 50 years of London development.

This interactive data visualization includes changes in population, expansion in transportation, energy, education and many other factors that will impact the city growth and change.

We created map-based tool which allows stakeholders to explore the city growth and change, using data collected by Greater London Authority (GLA). The tool consists of four main parts: interactive map, timeline, filters, and additional data visualizations. The map was built with esri API, and shows the most important information for the users. It also allows them to filter by borough, or hand-drawn polygon.

Timeline shows the time spans of scheduled projects, and allows to filter years which are interesting to the users, upon selection, map and other filters change.

Filters on the right side show the most important information for current view, and adapt to data shown on map.

Data visualizations on the bottom extend user information on selected view, giving them more context.

This tool is extremely interactive, a change in one filter or selection propagates through the whole application, allowing users to drill down and select the most interesting context for them. It was built with React, Redux, esri map API, and D3.

You can play with the live version of the application here: http://maps.london.gov.uk/ima/

I collaborated with Variable studio to create visualisation showing all Android device users, for the opening I/O keynote delivered by Sundar Pichai on 2015 Google I/O.

The visualisation was seen by 6000 attendees, and over 1.8M viewers online, and was displayed on 27k curved display, with over 17 million pixels covering 5m x 162m long canvas.

We started by exploring the dataset, using variable in-house pex library, writing a lot of small exploratory apps to test the ideas and see what we are dealing with. We were also researching ways of rendering on such a huge display, in order to keep artifacts away.

We created a lot of custom tools for this project, most important ones being a timeline which helped us to time things properly and multiple views into the final screen, so we could test if element sizes and speeds are working.

We were able to run the final code with smaller sample dataset in Google Chrome. Final rendering was done in Plask in less than 4 hours, thanks to JavaScript and WebGL we could quickly iterate and simply share preview visualisation with Google Data Arts team. Almost all of heavy-lifting was done with pixel shaders - including particle movement, using heavily modified flow-field, particles landing on the globe, and final globe unfolding.

I've built an interactive installation for 20th Kids Art Biennale, for exploring the city of Poznań through field-recordings made by children.

The map became an interface for mixing different sounds together, letting children and grownups create ad-hoc experimental audio compositions.

Installation was created with using web technologies - mainly React and D3 - which allowed us to use the same codebase for on-site hardware (55-inch touchscreen table), and online version: fonomapa.com

Software was built entirely from scratch, one of most interesting techniques we used was rendering vector tiles with D3, which allowed us to control the design of map in very fine detail. Map was designed to give only a little help to the user, moving all the attention to the sounds.

We also created custom CMS for easy sound upload and edition, with backend powered by node.js and mongodb.

Kinect2OSC is a small application for receiving and transmitting data from Kinect 360 through OSC.

It was build with openFrameworks and is available as open source: szymonkaliski/Kinect2OSC

LoopPI is a standalone four track audio looper made with Raspberry PI, ChucK and Node.js.

Each row of controls consists of two buttons and two potentiometers. First button toggles recording on and off, second one clears the loop, first potentiometer controls the loop volume, and second one controls feedback. Each loop is 8 seconds long with independent volume and feedback controls.

Audio processing part is written in ChucK (C-like language for audio), interface (buttons, potentiometers and LCD) in node.js.

This project is available as open source: szymonkaliski/LoopPI

In 2016 LoopPI was updated with recent Raspberry PI and Wolfson audio card for better sound quality: szymonkaliski/LoopPI2

EEG2OSC was created for Rafał Zapała's Sensorium project. It was used to pass data from Emotiv EPOC EEG Headset to Max/MSP.

This tool was created with Objective-C, and uses Emotiv "Premium" SDK (for access to raw EEG data), and MetatoneOSC for OSC communication.

The code is available as open source: szymonkaliski/EEG2OSC/

I collaborated with talented Rafał Zapała on his Sensorium installation, based on the idea of biofeedback from the viewer.

I created software for combining different signal sources - including EEG, GSR and pulse sensor, which let Rafał build his interactive composition, meant to be played on human emotions.

For EEG data we used Emotiv EPOC and written a data transceiver which outputs OSC messages - it's open source and available for free: szymonkaliski/EEG2OSC.

Pulse sensor and GSR were using e-Health Arduino platform: https://www.cooking-hacks.com/documentation/tutorials/ehealth-biometric-sensor-platform-arduino-raspberry-pi-medical

Both hardware sensors were outputing OSC at the end of processing pipeline, and Rafał wrote his own Max/MSP code to generate music.

I collaborated with talented Marek Straszak on interactive audio art piece, built for Art+Bits festival in 2014, and later exhibited at WRO Biennale 2015, the biggest Polish new-media art biennale.

The installation consists of five steel deconstructed icosahedrons, equipped with Arduinos and solenoids, which can be wirelessly controlled by a touchscreen interface. Solenoids hit the steel frame making sound, and because all of the geometries are different, the frequencies they make differ too.

The heart of the installation was a minimalistic touchscreen-based UI, which allowed visitors to sequence solenoids hits.

I created a generative logo, working in close collaboration with La Moulade design studio. We started from scratch, with an idea of organic movement and simplicity in mind.

First explorations started with metaballs, playing with different styles and ideas:

A brief episode of highly sci-fi looking blobs:

It appeared that metaballs resolution was a big issue, so we settled on using sphere vertex displacement:

Colors were an important part of final visualization, we experimented a lot with different ways of randomizing, and mixing shades:

Nodation is an experimental take on creating music using graphs. It was created during my residency at Fabrica in Fabruary 2014.

Node distance from center is translated to pitch and connecting nodes creates melodies:

• To create playing node, click anywhere on the screen.
• To connect nodes, hover over one of them, and drag line to another one.
• To remove nodes, drag them to bottom red part of the screen.
• You can save your creations, and share output URL.

Project was created using node.js and mongodb for simple server (saving and loading of graphs), and Pex for frontend.

Live version is running on heroku: http://nodation.herokuapp.com, and code is open sourced: szymonkaliski/nodation.