At Dynamicland, buildings are computers

Subtitle: This computing research lab is reinventing computer programming.

Original: At Dynamicland, The Building Is The Computer (2019)

Walking into Dynamicland, Oakland's computational research lab and public computer studio, the first thing I noticed was a series of projects spread across several workbenches and kitchen countertops. 3D printed math sculptures, plush toys, children's toys disassembled into parts, and a MIDI keyboard connected to a Raspberry Pi. There are enough art supplies here to rival any kindergarten classroom: markers and crayons, glue sticks, plasticine. On the walls are huge posters titled "Annotation of the Celera Human Genome Assembly" and "Outline of the Discussion Leading to Maxwell's Equations." Bret Victor, the engineer and designer who runs the lab, loves these informative posters because they free us from the constraints of glass rectangular screens.

In the corner of the building, there is a library. The books are STEM-focused, but delve into the arts and humanities. A copy of A History of Engineering & Science in the Bell System sits on the coffee table, and a poster on one wall reads "Every Representation of Everything (in progress) , showing examples of different musical notation systems, sign language, mathematical expressions, chemical notation.

Dynamicland is a nonprofit that lives between academic research labs and Silicon Valley startups, between physical and digital, between the distant past and future of computing. Researchers here are inventing a new medium of computing.

What is a computing medium - how was it invented?

Before we get to that, let's take a look at an ancient medium: maps. Reading a map is a complex and uniquely human skill that is not at all obvious to small children. You pull away from yourself and fly into the sky, abandoning the perspective you have been accustomed to all your life. Your imagination turns squiggly blue lines and shades of green into creeks, mountains and forests seen from above. Put all of this together in your head and you can picture your surroundings.

And you don't have to be a professional cartographer to draw a map. Using crayons and napkins, a child can draw their neighborhoods, including secret hideouts, friends' houses, parks, and ice rinks. That's the beauty of physical media versus digital: You can create it with anything you can find. Draw on a napkin. Draw on the sand with a stick. The medium is open for a wide range of improvisations.

However, digital media tend to be rigid. Commercial applications force us to work in a medium that is strictly regulated by a small number of designers. For example, Instagram only has three brush tools and five fonts for drawing on photos. Creativity, flexibility beyond this range is possible, but requires more effort.

Of course, every medium has its limits. That initial liberation can eventually become a cage that limits our range of expression and thought. Because every medium has limitations, there is always an opportunity to invent new mediums that expand our ability to understand the world around us, our ability to communicate, and our ability to solve the major problems of civilization.

Much of what we use today is an extension of the analog past: emails, e-books, and digital photos are more or less direct continuations from physical letters, books, and photos. This trend has trickled down to hardware products: we replaced $0.05 pencils and $0.005 paper with $200 digital pens and $1,000 tablets. By bringing the ways of using pencil and paper into the digital realm, we fool ourselves into not discovering entirely new ways.

That's what Dynamicland has been looking for: new approaches. A medium that can only be developed on the basis of computers and electronics.

The invention of new media is a recurring chicken-and-egg dilemma; a long-standing push-pull relationship between the medium and the work created by the medium. Xerox's Palo Alto Research Center (PARC) invented the Smalltalk programming system in the 1970s, and each 2-4 year research cycle followed the same steps:

1. Build applications in the language
2. Based on this experience, redesign the language
3. Build a new system based on the redesign.

Over the course of a decade, the researchers went through five such cycles before releasing the first public version of Smalltalk. It became a pioneering programming language, and the most popular modern programming languages, such as Python and JavaScript, absorbed the main ideas that started with Smalltalk - such as object-oriented programming and reflection (reflection). Smalltalk's graphical user interface became the basis for the Macintosh interface.

The Smalltalk research team was led by computing pioneer Alan Kay . A man of lofty ideals and technical expertise, Kay is a natural teacher, full of charisma and full of great stories and anecdotes about innovation. His life's work has been to create a dynamic computational medium -- and dynamic literacy -- for children. Kay built an early prototype of a notebook/tablet from cardboard in the late 1960s, decades before actual production, which he called " a personal computer for children of all ages ."

For decades, Kay has moved between the advanced research departments of major companies such as Apple and Hewlett-Packard, and most recently he has his own research facility , Viewpoints Research, based in Los Angeles. In his presentation, he argued that the conditions for critical long-term research that can truly benefit humanity rarely occur within companies because of the market's obsession with quarterly financial statements. To make real progress, research needs to be protected from its influence.

In 2013, Kay began working with Vishal Sikka to create a new lab to reinvent computing in the spirit of Xerox PARC. Vishal Sikka is committed to Kay's vision, and as SAP's CTO, he has the resources to fund it. The lab is called The Communication Design Group (CDG) . Kay hired three principal researchers: Dan Ingalls , Vi Hart, and Bret Victor , and provided them with space and time in San Francisco.

For Bret Victor, whose work at Apple influenced the iPad and Apple Watch, the golden age of long research at Xerox PARC, Bell Labs and others in the 1960s and 1970s seems to be over. But when Kay approached Bret Victor and other researchers to form the CDG, it was clear there was a torch to pass, with each researcher putting their own perspective into it.

Inspired by Kay's vision, Victor quickly discovered that inventing new dynamic media was his natural job. Over the years, Victor has come to the conclusion that there is an urgent need to get rid of many of the fundamental assumptions about what programming is and who has the right to program; these assumptions had been sunk in the sedimentary layers of technology 40 years ago and have barely changed since.

If we want a future where everyone can program as easily as drawing a map on a napkin, and not just professional programmers have access to the full power of computing, we may need to reimagine programming itself.

Traditionally, programming languages exist in a holy trinity: language, tool, and operating system. Trinity makes programmers masters of a common toolset, available in multiple languages. As C++ creator Bjarne Stroustrup wrote in "The Design and Evolution of C++", "For most industrial users, the programming language is just a much larger This need is extremely important as a cog in the machine." He attributes C++'s widespread appeal to its ability to fit into the Trinity. This is very helpful for professional programmers when the languages work together well. The desire for interoperable languages and common tools is the foundation of the Trinity, and any language that tries to break through is unlikely to become very popular.

As a result, most professional programmers today edit text files every day in an 80-column-wide command-line interface originally designed in the mid-1960s. And most people don't even question that. But there is a subculture of programmers, naturally centered on Victor, who believe that programming is in a dark age because of a near-universal commitment to the Trinity.

We need to break out of the Trinity programming system that feels dynamic and fluid, and is closer to what most people care about. Spreadsheets offer an alternative: they provide an environment that provides instant feedback, where code and data coexist in a graphical interface with a language well suited to common tabular data problems. Scratch is a popular programming system for children developed by MIT specifically for play and creation.

Learning to program is all about learning to solve problems while exploring the special creativity that computers and software have. Programming makes information a living thing, able to flow through the system and adapt to various processes and models. It allows information to dance. It can be more beautiful, more playful, more human, and more accessible than what the Holy Trinity has to offer.

In 2013, CDG worked with Vishal Sikka of SAP and his team to create a layer of isolation between them and SAP so research could proceed. Many of the decisions they make require exceptions and workarounds to SAP policy: they want their own office space, they want researchers to retain intellectual property, and so on. In a company the size of SAP, even with the support of a CTO, cutting red tape was tiring for Victor, who took on a lot of work. At the same time, he had to figure out how to create a research lab, find and design a space, build a team, and plan the research itself.

The work paid off, and 2014 was a fruitful year. Victor hired four researchers to work with him, and he gave them autonomy to work on their own personal projects within a broad agenda . Early projects include Toby Schachman 's ShaderShop , which allows people to create low-level graphics programs visually rather than code; Glen Chiacchieri with PBS The documentary Eyes on the Prize conducted a video experiment .

The researchers think they have at least 5-7 years of funding. With ample funding in the bank, this research can be exploratory or playful. No pressure to get anything to market. Researchers can let go of the fast-paced productivity manifestations that pervade startup culture and work at their own natural pace. They can follow intuitions that are unlikely to have any outcome. They can spend long hours reading quietly and doing deep work in a direction of their choice undisturbed. And, when inspiration strikes, they can work overnight — or for weeks on end — working on a new prototype.

All this space and openness has created at least one existential crisis. For Glen Chiacchieri, a young software engineer recruited from the MIT Media Lab, working for Victor seemed like the ideal job at first. But knowing he could do absolutely anything, he deeply questioned his values. This created a lot of anxiety and eventually depression, and in his second year in the lab, he spun around in despair for months. "No one told me what to do, I had to decide for myself what was meaningful in my life. Because I took my job very seriously, the process was very difficult for me," he wrote in a recent article. road. Eventually, after a series of psychedelic trips with psychedelics and existential conversations with colleagues, Glen Chiacchieri opted out of the tech industry to become a psychotherapist.

In the summer of 2014, just a few months after the CDG was established, Vishal Sikka abruptly left SAP, hampering the exploratory opening of the lab. After losing SAP's internal champions, CDG's long-term future doesn't look clear.

Kay largely shields researchers from this interference. At the time, Victor's team was creating a prototype system called Hypercard in the World , which would allow people to attach hyperlinks to physical objects.

Victor and researcher Robert Ochshorn turned all of the lab's past work into a "Hypercard in the World ." Point a laser pointer at a project or paper listed on a large "index poster," and the adjacent display shows more information about the project. Another project by May-Li Khoe , called Serengeti, used animal decoupage to create a dynamic diorama of desert animals. Visitors can point to an animal with a laser pointer to learn more about it.

In the late summer of 2014, CDG hosted a Game Jam, which was a truly shining moment of fast-paced collaboration and productivity. Friends from the lab came over to do more than a dozen "world super card" projects. Glen Chiacchieri made a party game called Laser Socks , using a laser pointer, a projector pointed at the floor, and people jumping around in socks.

Over the next year, more exploratory prototypes were built. “I would do these weird art projects,” says Toby Schachman. He hosts a Mirror Hacking Workshop , inviting people to make sculptures out of laser-cut mirrors. At the time, it seemed unrelated to research, but after thinking about it, he gained insights into collaboration, the power of eye contact, the use of physical materials, and allowing people to see what others are doing. He began to ask, why can't computers be like this?

At the same time, the relationship with SAP gradually broke down, and by early 2016 it became clear that CDG needed a new home. Around the same time, Victor made it clear to his team that they needed to come together and build a single system, rather than primarily working on a single research project. Several researchers were reluctant to get involved and left.

By May 2016, Kay had attracted Sam Altman , president of Y Combinator. They created HARC (Human Advancement Research Community) within the YC Institute and absorbed CDG researchers there. Sam Altman graciously agreed to fund HARC out of his own pocket while they waited for other pledged funds to arrive.

This arrangement continued for more than a year. In July 2017, just a few months after HARC moved into a beautifully renovated building in Old Auckland, Sam Altman abruptly withdrew funding for the lab.

It is unclear why he withdrew the funding. In his annual letter to Y Combinator in February 2017, he said the work of Victor's lab "remains one of the new technologies I'm most interested in." But a person close to Sam Altman told me that by July his excitement had shifted from HARC to OpenAI, another YC research project, of which he is now CEO. After that, Kay went to London and the research group disbanded.

However, Victor's team had one thing that kept them going: just two weeks before they learned that HARC was shutting down, the team's new programming system -- Realtalk -- had taken its first steps. Running at about one frame per second, it's more of a crawl, but it's a very exciting crawl.

The idea behind Realtalk incorporates all the lessons learned from past projects, such as "Hypercard in the World." Researchers - Josh Horowitz , Luke Iannini , Toby Schachman, Paula Te and Brett Bret Victor, and Virginia McArthur - It took a year to design the new system and two months to put it into operation.

Encouraged by the potential of the new system, Victor's research group chose to go out on their own, take over HARC's Oakland space, and start fundraising for themselves.

In a talk at Harvard last year, Victor laid out three main design principles from the research and incorporated them into Realtalk:

• The medium should be public and accessible. People should learn and collaborate through perception, rather than assuming a single, isolated user sitting in front of a laptop with a keyboard and mouse.
• The medium should make people think with their bodies, because we are not just fingers and hands.
• The medium should amplify people's agency and unleash their creativity; not become an application with limited functionality defined by companies and imposed on people.

Rather than simulating things like paper and pencils in a computer, Realtalk gives computational value to everyday objects in the world. Architecture is computer. Space is a first-class entity—the cornerstone of computing. Digital projectors, cameras and computers are mounted discreetly on the ceiling, creating spaces on tables and walls for projects and collaboration. Most software is printed on paper and runs on paper. But the deeper idea is that when the system recognizes any physical object, it becomes a computational object.

Several researchers have never created such a complete, low-level system before. This narrowed their focus and gave the team a sense of consistency that allowed them to start fundraising in time.

With only a few months' worth of cash in the bank, they made a tradeoff. Instead of pursuing conceptual perfection and deep flexibility, the team focused on polishing their stuff for funders.

Today, the Realtalk projects have taken up the bulk of the building - every project is composed of paper and light. Tables and walls are illuminated by colorful comets, rain, frogs, octopuses, puzzles, charts, clocks and maps. A music arranger called Beats of the World by Paula Te lets you design rhythmic loops with symbols. GeoKit by Omar Rizwan is a giant interactive map.

I sat at a desk and pointed my keyboard to a piece of paper labeled "Code editor" and an editor appeared.

I press Control-P and a laser printer spits out a page with my program written on it.

The "Hello, World" page looks like this:

Put this page on any surface and you'll see its output projected on it:

As long as it's on the table, it's running. Flip it over and it will stop.

Paper can be cut, flashed, stamped, torn, taped, stapled and scribbled. It can be glued to a spinner and made into a miniature book, or folded into origami to bounce around on a table. This is a good prototyping format. Kids love it. Everyone loves it. It instantly sparks new ideas.

Realtalk projects are implemented by writing one or more programs on paper and arranging them on a table. Papers can easily communicate with each other, making creation and interaction simple.

In addition to paper, there are points. Computer vision algorithms identify chocolate chips, small signs, and even painted nails as dots. Want a slider control? Use a marker to draw a straight line, draw a dot on the paper, and swipe with your finger. Or stick a dot on the end of the popsicle.

Print out the entire source code of Realtalk and stick it on a few scrolling whiteboards. The tools developers use for printing and debugging hang from the boards, just like in a carpentry workshop. Items are tucked into binders and plastic boxes, labeled Dinner Party Games, Rainbow Canvas, Shape Loops, and Radial Animation. People learn by example, reading pages posted around the space, flipping through a ring-shaped tutorial magazine that can run on any surface.

The above code runs continuously when you put a page on the table. Feedback is instant as you make changes. There is an indescribable feeling of dancing with code. It's easy to find the satisfaction of making rapid progress. It maximizes the fluidity of ideas. I'm often amazed at the new ideas that emerge from bugs and weirdness. I let go of "what should I do?" as I head in the direction I want, considering the unexpected surprises of the process.

Realtalk programs are constantly being remixed and delivered. They become physical memos and are reprinted, copied and modified. I achieved this with a simple Magic Eight Ball page attached to a spinner. I wrote it down without thinking, put it on the table, and when I came back a few weeks later, I found several mixed versions of it in the space. Pages with emotional power or general utility seem to proliferate.

The "Publicity and Accessibility" principle is where the lab still has a lot of work to do. This is a difficult problem. Most of the major projects on display were done by individual researchers, not a group. Several of the researchers were programmers who grew up coding themselves, a habit that was hard to break. It's tempting to code on your own.

It makes sense that they are selective about who they invite to the lab and when (and how) to share the research itself. They are sensitive to the contamination of their language and way of working by Silicon Valley and brogrammer culture. There is also a tension between, on the one hand, to allow research to be carried out in a safe laboratory environment, and on the other hand to create a community space where the medium can interact with a wide variety of people during the design process comminicate. "We have to open up this research to get more people involved and help us shape it so it doesn't just benefit those lucky enough to be programmers," says Paula Te. "As someone whose non-white, non-male perspective has been ignored, my goal is to make those who are not represented by mainstream culture a part of inventing new mediums."

One factor that encourages an open community is that Dynamicland is the only place you can do Realtalk projects. I can't do this project at home alone. Realtalk doesn't have GitHub. My time at Dynamicland feels precious, and that preciousness seems to boost my creativity. Maybe that's how college computers felt before the PC era.

Is Realtalk the future? No, in the bigger picture, it's just an iteration. Here's a suggestion: maybe by going back to the drawing board and redesigning something deeper, we could escape the dark ages of the command line, and computing power itself could become more accessible and shared.

Victor's dream is to be able to experience an entire scientific paper, or an entire global supply chain, in a room powered by computation. Explore data richer and deeper than a single screen. At the most existential level, he hopes the research may help avoid human extinction. If we can understand the complexities of our world more broadly, we will be in a better long-term position to mitigate risks that could threaten civilization.

In early 2019, on the coldest day of the year, I walked to Bret Victor's apartment in Cambridge, Massachusetts, to meet him. I walked past IDEO's Cambridge offices between Harvard and MIT. Design firms like IDEO sell a style of research that promises new ideas to blue-chip clients. The façades are painted with geometric shapes and bright colors, symbolizing a modest amount of creativity. As said above, we are playful, but serious. We'll help you safely enter some new creative territory. We bring the best in design thinking and rapid prototyping. We promise not to be too wild.

By contrast, Dynamicland seems to value complete intellectual freedom. When I first visited the lab at an open house in early 2018, I thought of the Children's Television Workshop in its heyday: counterculture, research-backed, subversive, experimental, sometimes brilliant, But always underfunded. Several tireless inventors worked around the clock. Open any drawer and you'll not be surprised to find a pile of rolling toy eyeballs inside.

Victor left the Bay Area last fall and is taking a sabbatical, recovering from years of funding and managing the lab. He greeted me at the front door with a warm smile. Although Victor grew up in the Bay Area, he seemed at home here, just steps away from MIT's campus.

He was drinking from a bottle of water as we talked. He has a vast knowledge of computer history, and he loves telling those stories. He told me about the hacker culture that emerged around the TX-0 computer at MIT in the mid-1950s. The TX-0 is right in the lobby of the Model Railroad Club , and some railroad fans find they can use the computer alone late at night because researchers often oversleep and miss their use. That's where the first known word processor was born - it was called the "Expensive Typewriter" because everyone thought it was ridiculous to use a $3 million computer to write term papers of.

As a lifelong software engineer, listening to Victor's story made me a little embarrassed because I'd never delved into the history of computer science very deeply.

We ended up chatting for hours. When he started telling me stories about the lab, I sensed that he was always looking for a compassionate listener. Some kind of catharsis from six years of roller coaster experience.

He seemed tired. CDG is his first management role. For a prolific creator, it's a huge transition that requires a lot of energy and personal growth. He must have a context-making background (Dynamicland) and contribute to the context itself (Realtalk), while also dreaming - but having little time left to make - projects in the context .

This is clearly his life's work. He often has a sly smile as he tells his stories about Dynamicland, looking like a little boy who knows he's done a very smart thing. We talked a lot about the context-building work he did, and he said he was trying to use Dynamicland as a biological isolation facility. He worries that half-baked ideas that entrepreneurs have stolen from research labs have done too much. "A massive, superficial treatment of a deep idea, after which it's impossible to see the deep idea," he said. (The thing about taking a deep idea and making a mass-produced, superficial treatment of it, is that after that point it becomes impossible to see the deep idea.)

He told me the story of Steve Jobs taking the GUI idea from the Smalltalk team and applying it to the Macintosh. In Smalltalk, a graphical user interface is used to program the computer. The original concept of object-oriented programming was entirely graphical—objects on the screen represented objects in the program. This is a very profound idea. But on the Macintosh, the GUI simply cannot program the computer. To bring the first Macintosh to market, Jobs made a superficial copy of the deep ideas of Smalltalk.

Victor's appreciation for deep thought is his greatest strength and his greatest weakness. This gave him the purpose and energy to create the Dynamicland environment and allowed him to work on his deep ideas for a long time, bringing them to life in beautiful ways. The risk, however, is that nothing is considered done and therefore nothing is shared. In other words, every sharing is temporary. He seems to want to be understood so badly that his expression of ideas is never deep enough.

Usually one person doesn't play both an inventor and a performer. Les Paul invented the electric guitar to bring more warmth to his smooth jazz, and he never foresaw how Jimi Hendrix would use it. In his 1996 diary , A Year with Swollen Appendices , Brian Eno wrote: "The overloading of the guitar means breaking the frame of the device - doing something out of your control - that's for a person 's expressive palette adds a whole new dimension, because now one can juxtapose what one can control with what is 'cannot'."

Victor is Jimi Hendrix who plays Les Paul, and there's a tension here that makes him well-suited to endure the cycle of making and breaking as the medium evolves.

He doesn't ease the endlessness of work by looking for market potential like a startup founder. The pressure on nonprofit research labs is different: they need to present research in a way that invites collaboration and funding. But Victor and his team were exhausted on fundraising, which required a lot of time and effort. Also, he has his own sense of when a research cycle ends, and he doesn't want to let funding decide.

However, this may be all he needs. The ups and downs of funding are an important factor driving Dynamicland's development. Laboratories benefit from external accountability to control very personal creative research work, and Dynamicland has a strong track record in creative output since 2013, although Victor doesn't seem to like being the focus of large demonstrations or presentations. The work is notable in the tech world. Dynamicland is loved by social media, and visitors can't help but post online about how awesome Realtalk is, so this project has a life of its own on the internet.

But how will this research reach a wider audience? Can its influence be engineered or even manipulated, or is it completely out of their control?

"I don't know what the solution is for a larger audience," he said. "Maybe a hundred years. It took a hundred years from the invention of the printing press to when books became part of popular culture."

Later, he told me, they had a plan. In July 2019, Dynamicland started developing the next iteration of their programming system - Realtalk-2020 .

From the perspective of venture capitalists, technological innovation occurs during the evolution of the entire entrepreneurial ecosystem. There have been many cycles of life and death and acquisitions. But if every startup operates under the exact same constraints of runway and product-market fit, that inhibits the kind of deliberate creativity that Dynamicland fosters. We need more organizations like Dynamicland, occupying the space between startups and academic research labs. These places are designed to make dreams come true over a longer period of time.

Back home in San Francisco, I couldn't help but wonder about the intrinsic motivation the researchers brought to this work. If research is very open, then perhaps a researcher's work is not "done" until they have solved their core dilemma. Glen Chiacchieri's desire to help people feel more happy led him to training in psychotherapy. Toby Schachman cares so much about eye contact and fun collaboration in Realtalk because he wants to feel more connected. Paula Te wants to create a medium that truly respects diversity because she wants to feel a greater sense of belonging in the field of computational research.

What about Victor?

Over thousands of years, empirical evidence and bottom-up scientific discoveries have gradually taken their place from traditional top-down religious teachings. What we think of as unknown or mysterious is shrinking, even if the size of the universe means the shrinking form is ∞-x, where ∞ is the vast unknown and x is the totality of human knowledge.

On the surface, Victor seems most interested in expanding x following in the footsteps of great scientists and researchers. But what was his inner motive?

The desire to expand consciousness. More "into the world". Experience the full complexity of reality. Explore the creative potential of white space. Get rid of the limitations of your peers' thinking. Break through the cage of appearances.

Coincidentally, these were the main themes of the Indian spiritual philosopher Krishnamurti 's treatise on spiritual enlightenment in the mid-20th century.

Krishnamurti struggled to speak on behalf of the ∞ side of the divide. He had a hyper-rational view of spirituality, he distrusted religion and ideology, and he advocated rigorous empirical self-inquiry. As Viktor explores the representation, simulation and modeling of the external world, Krishnamurti asks us to methodically and internally let go of all representations and concepts.

"Can [the mind] be free and empty?" Krishnamurti wrote. "It can only be empty if you understand all its projections and activities, not intermittently, but day in and day out, moment by moment. Then you'll find... the state of creative emptiness is not something to cultivate — it is there, it comes secretly, without any invitation, and only in that state is there a possibility of renewal, of newness and of revolution.”

Victor wanted to experience the revolutionary ∞. He appears to be on a spiritual quest for what Alan Kay calls "a kerpow." A beginning to a new dimension. For somewhere beyond the land of thought altogether beyond any deep thought, there is a rich, boundless territory that has never been mapped, despite our thousands of years of trying.