I've been also continuing research at Ink&Switch around the topic of "programming by drawing" for a tablet form-factor (trying to bring some of the Dynabook ideas to life). Latest spin-off work includes in-house Lisp with a projectional editor. This work is not public yet, but if you're interested in these topics, I'm happy to give a demo on a call.
At flow/control, we finished a bit more conceptual consulting project with Clay. We worked on exploratory research on integrations and enrichments — basically how to make Clay a central point for connecting various web applications through a end-user programmable spreadsheet.
Most of my time over the last three months went into Ink&Switch, where we were ramping up on a new project, culminating some of the research I've been involved in for the past year. We're also finishing (hopefully!) a big "real-life" project, so there wasn't that much time for side-explorations, but I managed to squeeze in two smaller things that might be interesting to some of you.
Running incomplete programs with type checking is still an open research topic, one of the most famous examples being Hazel, which not only allows you to run unfinished programs, but can also (partially) type check them! It's definitely worth looking into, but: one — it's a research project, and two — a totally new universe to adopt (which you probably shouldn't do right now, because of the first point).
Reference material and prior art is an important part of research and design work — and since research and design is what I now spend most of my time on, it's important to me to deal with all the different snippets I keep on collecting. I use two online platforms for storing things for later: Pinterest where my main visual references end up, and Pinboard for links to articles and interesting websites. I also take a lot of screenshots and store them all, permanently, in a folder on my hard drive.
During the summer of 2020 I played around with an idea for research-focused web browser, embodying some of the concepts from Browsing vs Searching note — browsing being an open-ended divergent activity, and searching understood as information retrieval.
Unfortunately, in practice, I don't think having the full history is that useful. Yes, it's sometimes good to know how you ended up somewhere, but I think what's most valuable about "research" is the synthesis part — grabbing parts of larger wholes, rearranging, recombining, thinking with the material. A small step in this direction could be persisting scroll position or maybe selection, and making the history editable — allowing users to remove dead ends, add notes, etc.
As a small aside (slash ramble), "programming" is in quotes, because I really don't like that word for describing this sort of work. I have some early illegible thoughts about this in End-User Programming vs Programming note, but the gist of it is that what I've been after in this research track is "programming" as a tool for understanding, and working with, dynamic systems — same way as a numerals and mathematical notation are tools for working with mathematical ideas. It now seems to me that a lot of Future of Coding work (including most of my own!), is about making lives of "expert symbol manipulators" easier — which is definitely worthwhile — but this year I started to understand that there's a lot of value in making "working with dynamic systems" accessible to everyone, and that might need another approach than iterating from where we are right now.
Once I'm back, I'm going to spend most of my summer as a "Researcher in Residence" at Glide, prototyping a programming environment for building custom UI components that fit with the Glide's aesthetics. Just a quick reminder — I do consulting, specializing in digital tools and no-/low-/feature-of- computing research, design & development. Reach out if that sounds interesting.
As you might remember, I spent this quarter doing a research residency at Glide, where I prototyped a programming system for building custom components, focused on composability, re-use, and liveliness. We're working on a longer write-up, and I'll let you know when it's out. In the meantime, I recorded a short walkthrough to give you some sense of where we ended up, that you can check out below.
Most of my free time this quarter went into preparing for my Strange Loop talk about the research I've been doing at Ink&Switch. From what I heard the talk went pretty well though, and I enjoyed all the hallway conversations that it spawned. The video will be out sometime in the upcoming weeks on the Strange Loop YouTube.
After four years at Ink&Switch, two of them as a Principal Investigator, I want to turn the dial from pure research, to more industry-oriented work.
Finally, I wrote a long article documenting my research into end-user-programmable UI-builder at Glide, which includes a live version of the prototype — go check it out. If you want to see it in action with additional commentary, I recorded a short informal demo in the last newsletter issue.
The only technical preparation I went through was creating a slide deck for the portfolio reviews. A ton of my projects are online, but I wanted to also walk through the process, give additional context, etc. I strategically picked two projects: Crosscut which shows the research-y and team-lead-y side of me, and Glide's Code Components which was the most recent, most production-resembling, solo work.
Going from 10-ish person research lab to 100-ish person startup definitely is an adjustment. For one, it's impossible to know about everything that's going on. Two, small details matter in addition to the "big vision". With exploratory research, it's often a good idea to hand-wave at things that would be solvable with additional engineering time, here, you have to do the "additional engineering" too, which changes the pace (and focus) of the work quite a bit. And three: you actually have feedback from real users and the outside world, and while sometimes emotionally difficult, it dictates a real goalpost to go after.
We hit this earlier on in Inkbase where interacting with a single item was nicely visualized with inspector panes, but groups of objects were not. We tried to explicitly solve this in Crosscut, but the solutions were far from ideal. I also hit this in my research at Glide where displaying lists of values is the main thing you do.
Designing and implementing tools with a good cognitive fit for user and task is an exercise in removing gratuitous difficulties and finding familiar, expressive interface metaphors. Because there is an existing way the task is being performed by existing users (...), it is possible to invent a new tool idea and quickly test the idea with users to see if it in fact makes a task easier. Often users are able to state with confidence whether or not they believe a new way of doing a task will be easier and more efficient based solely on hearing the idea, without having the new tool implemented. However, designing and implementing a dissemination strategy with a good social fit for the user community is far more difficult. A good social fit means users eagerly give up an old tool and adopt a new tool. The time constant in social fit (new product adoption cycle) is typically much longer than the time constant for cognitive fit (duration of an authoring task). New tools that are free to try, easy to learn, similar to existing tools, demonstrate rapid payback, small and fast are likely to spread rapidly in a community of users.
— ATG Education Research - The Authoring Tools Thread - Jim Spohrer
Non of our recent work would happen without Figma. From boards with brief breakdown, desk research, data discussions, concept drawings to WIP screenshots, branding guide and UX user journeys. We bring visual thinking to the way we both design and code.
— Marcin Ignac - https://twitter.com/marcinignac/status/1200022203728838656
I understand, then, why researchers flock to the safety of institutions. Imagine studying something that nobody else is studying, for reasons you can't really articulate, without knowing what the outcome of your work will be. For the truly obsessed person, the need for validation isn't about ego; it's about sanity. You want to know there's some meaning behind the dizzying mental labyrinth that you simultaneously can't escape and also never want to leave.
— Nadia Eghbal - https://nadiaeghbal.com/independent-research
Perhaps a more subtle practice I've found helpful is to find situations where the desired outcome is considered unremarkable.
I like being in a PhD program because "produce novel, high-quality research" is just what everyone is doing, not some cool special thing.
— Geoffrey Litt - https://twitter.com/geoffreylitt/status/1375116106734702594
The development of a program is not linear. Programmers neither write down a program in text order from start to finish, nor work top-down from the highest mental construct to the smallest. They sometimes jump from a high level to a low level or vice versa, and they frequently revise what they have written so far (...). For the purposes of programming support, that is all we need to know. Although the causes and nature of deviations from top-down development have inspired much research, the implication for a programming environment is quite straightforward; the order of working should be left to the programmer, not prescribed by the environment.
— Usability Analysis of Visual Programming Environments - A "Cognitive Dimensions" Framework
(...) purposeful practice in a nutshell: Get outside your comfort zone but do it in a focused way, with clear goals, a plan for reaching those goals, and a way to monitor your progress. Oh, and figure out a way to maintain your motivation.
Purposeful practice involves feedback.
Generally speaking, no matter what you're trying to do, you need feedback to identify exactly where and how you are falling short. Without feedback - either from yourself or from outside observers - you cannot figure out what you need to improve on or how close you are to achieving your goals. (...) This is a fundamental truth about any sort of practice: If you never push yourself beyond your comfort zone, you will never improve. The amateur pianist who took half a dozen years of lessons when he was a teenager but who for the past thirty years has been playing the same set of songs in exactly the same way over and over again may have accumulated ten thousand hours of "practice" during that time, but he is no better at playing the piano than he was thirty years ago. Indeed, he's probably gotten worse. We have especially strong evidence of this phenomenon as it applies to physicians. Research on many specialties shows that doctors who have been in practice for twenty or thirty years do worse on certain objective measures of performance than those who are just two or three years out of medical school. It turns out that most of what doctors do in their day-to-day practice does nothing to improve or even maintain their abilities; little of it challenges them or pushes them out of their comfort zones. (...) Generally the solution is not "try harder" but rather "try differently."
The design process is a sequence of expert activities that produces an innovative product. The artifact enables the researcher to get a better grasp of the problem; the re-evaluation of the problem improves the quality of the design process and so on. This build-and-evaluate loop is typically iterated a number of times before the final design artifact is generated
- but, the field-testing part seems very important; with Understanding Through Building I only "test" with myself, or even less, use the process itself to "only" understand the problem better without any real use for the artifact
in the case of the DSR, the proposal is among others to start from problems that come from the field (relevance) and to attempt to provide an artifact (solution) and to rigorously develop what amounts to prescriptive knowledge, while meeting the standards and norms of scientific research (rigor)
Design science research in itself implies an ethical change from describing and explaining of the existing world to shaping it. The ethics of research concern the responsibility of a scientist for the consequences of his research and its results. Even though it may be questionable whether any research can be value-free, it is absolutely clear that design science research cannot be. Consequently, the basic values of research should be expressed as explicitly as possible. Adapting Chua (1986), Iivari (1991) distinguishes three potential roles for Information Systems as an applied discipline: 1) means-end oriented, 2) interpretive, and 3) critical. In the first case the scientist aims at providing means knowledge for achieving given ends (goals), without questioning the legitimacy of those ends. According to Chua (1986), the aim of an "interpretivist scientist is to enrich people's understanding of their action", "how social order is produced and reproduced" (p. 615). The goals (ends) of action are often not so clear, and one should also focus on unintended consequences. A critical scientist sees that research has "a critical imperative: the identification and removal of domination and ideological practice" (p. 622). Goals (ends) can be subjected to critical analysis.
— A paradigmatic analysis of information systems as a design science
As you're typing them they're in a live editing preview mode, but when you press enter they get applied "permanently".
(...) you don't have to hit enter or a button to make it execute, and you don't have to undo if it didn't do what you wanted it to. The code is continuously parsed and interpreted as it is edited (as long as it is valid). The function-invocations register themselves with the editor UI, which draws these visualisations and keeps state for them as long as the editing session exists and the function invocation is not removed. This lets the user tweak the code and make sure it works, as well as tweak the selections etc. Once the user is happy with the result, they can commit the changes.
The traditional cognition approach assumes that perception and motor systems are merely peripheral input and output devices. However, embodied cognition posits that the mind and body interact "on the fly" as a single entity. An example of embodied cognition is seen in the area of robotics, where movements are not based on internal representations, rather, they are based on the robot's direct and immediate interaction with its environment. Additionally, research has shown that embodied facial expressions influence judgments, and arm movements are related to a person's evaluation of a word or concept. In the latter example, the individual would pull or push a lever towards his name at a faster rate for positive words, than for negative words. These results appeal to the embodied nature of situated cognition, where knowledge is the achievement of the whole body in its interaction with the world.
Some of my favorite work in tools for thought comes from idiosyncratic Twitter tinkerers. This group often produces fascinating work, but it's usually missing one or more of these steps. The most common pattern seems to be: a bricoleur identifies some powerful idea about a representation and design a prototype, but then fails to engage seriously with observing and deriving insight from the systems they've built. Sometimes this comes from technical barriers—the prototype is too quick-and-dirty to be used in a serious context, so their insight is limited. But I think there's also a cultural gap here, a missing research practice of careful, diligent observation and synthesis. Too often these projects have the flavor of "Look, I made a thing! Isn't it cool? How many people can I get to use it?". But the question they need to be answering is: "What powerful, generalizable ideas can we learn from this project? How should the next wave of systems build on this?"
— Ratcheting progress in tools for thought - Andy Matuschak
- designing the questions to ask after building a prototype (or even building the prototype to ask specific questions) is important part of the puzzle - building just for building sake lacks the feedback loop
These representations weren't mere scientific "discoveries". Each of them essentially enabled all subsequent scientific breakthroughs thereafter. A powerful new form of representation affects everything, forever.
— CDG Research Agenda - Bret Victor
- new powerful notations "stick around" and infect human thinking for generations
On various timescales needed for thinking and research
[Dynamic Authoring] The author creates the material by directly manipulating representations of behavior and data, instead of manipulating a structure. Manipulation takes place in the data domain.
I've advocated "learn everything and then forget it except for the perfume". This can create a mental space for thinking which will inescapably be helped by what we know - it's really hard to completely forget! - but in which what we know (mostly meaning what we believe!) is far enough away to allow us to feel things, listen to our subconscious whispers, and generally barge around.
— Alan Kay - https://www.quora.com/What-advice-would-Alan-Kay-give-a-curious-individual-to-improve-their-ability-to-think-and-learn-Is-there-a-place-to-see-his-library-%E2%80%94-every-book-person-and-research-he-has-studied/answer/Alan-Kay-11
One research study asked students to think about an important exam. Half of the students were asked to put in writing specific plans of what/where/when they would study. Later, all students were asked to do a word association test. The group of students that did not write any study plans produced more word associations related to studying because studying was still on their mind; the group who did write down their study plans did not exhibit a comparable bias during the word association test.
From technical side, FabFungus was a culmination of three personal research projects:
The focus of this research project was prototyping a UI-builder that fits into Glide's aesthetics, and provides a way to edit, and re-use, the visual components available in the system.
We started by exploring the dataset, using variable in-house pex library, writing a lot of small exploratory apps to test the ideas and see what we are dealing with. We were also researching ways of rendering on such a huge display, in order to keep artifacts away.
To learn more about the research, head over here for a long write-up covering everything from philosophy behind hand-drawn marks, to intricacies of the programming model.
Because my first time tracking system has been built before I've started tracking time, I have no idea how long it took to build it. The idea was pretty simple though, I wanted to store all the logs in a database, and be able to query and analyse them. The log consisted of starting and ending times, project's name that it belonged to and type — either personal or work, later another type, research was added, but I didn't use it much. I also tracked payments with date, amount and project name. Everything was held together by project name, so my tracking was only concerned about how I work on very specific things.
I enjoyed this month's exploration and learning a new language as nice break from tool-making research.
I was quite tired after last year, where I worked on monthly projects and pushed hard to publish something every month. I also knew that this year would be a bit different due to my personal life, so I had to plan accordingly. I decided to aim for working on one personal project every day, but without any tight deadlines; this could be reading a research paper, exploring some technology if I had time, or playing a bit of music.
Next time, I would probably aim for either fully research/learning related residency, or one where I know almost everything, and produce artwork. I've tried to combine both, and ended up only building the tools, without ever actually making anything bigger with them.
First week was full of learning in the most frustrating way — trying and failing over and over again. I still didn't know what exactly I wanted to do, and was exploring different techniques. I finally settled on using SDFs (signed distance fields) after short research on meshing isosurfaces.