This is the opening to my application in late 2012 for the Interaction Design MFA program at SVA, New York:
I was born in 1984, exactly four days before Steve Jobs took a small computer out of a bag and let it introduce itself to the world: “Hello, I’m Macintosh”.
While I was a little too young to appreciate this feat at four days old, I see being born in this period as intrinsic to my view of technology and our relationship with it. I’m part of the first generation to grow up with such consumer technology, yet I’m just old enough to recall life before it became ubiquitous. When we had to wait for each other at pre-arranged locations, and hope the other turned up on time. When we only logged into AOL for 10 minutes as we were worried about the cost. When we had to record our favourite programmes on VHS, then re-record over them once we were done.
Despite each new leap or paradigm shift nudging us into strange new worlds, technology has always felt like second nature. It feels ‘obvious’ or ‘inevitable’ in a way it appears not to for those only a little older.
What I meant by ‘obvious’ was, in a way, digital literacy. I’m part of a generation often described as digital natives. Yet as familiar as I was with using technology, and in working with technology prior to grad school, my grasp of how computers worked was anecdotal and limited. In the late 1990s I could leave a computer on all night downloading a rare Radiohead song and play it the next morning in RealPlayer. I once tracked down a hidden file to remove a Windows virus that prevented access to Google. But I couldn’t, as recently as 20 months ago, have told you what a Boolean was.
My personal journey from digital native to ‘code immigrant’ has been revelatory. From an acute fear of code, I now love nothing more than spending days immersed in it. And that transition provided a key inspiration for my thesis work. But it began with a different anxiety. One about the small machines we carry in our pockets and, increasingly, elsewhere on our bodies.
Over the last few years I watched the rise of mobile and smart devices with awe and fascination, but beneath my enthusiasm lay a creeping worry. Slick, closed, ‘magical’ devices reveal little of their workings, and create a more one-way, consumption-oriented relationship than many of yesterday’s computer pioneers dreamed of and advocated for. The technology industry’s drive to realize Arthur C. Clarke’s rule about sufficiently advanced technology has become an obsession.
This is not to that the rise of mobile isn’t an incredible feat of progress. Nor is it to say that making technology immediate and accessible is a bad thing. Quite the opposite on both accounts.
But is there a necessary relationship between ease of use and obfuscation, or have we simply decided that most people should be satisfied to use consumption-led devices and restricted tools?
The web contains a degree of 'unsealability', you can view the source of a website and look beneath the surface. Apps have no such affordances.
Alan Kay: I characterize what we have today as a wonderful bike with training wheels on that nobody knows they are on so nobody is trying to take them off. I just feel like we're way way behind where we could have been if it weren't for the way commercialization turned out.
Doug Engelbart: Well, strangely enough, I feel the same. It's part of the thing of the easy to learn and natural to use thing that became sort of a god to follow and the marketplace is driving it and it's successful and you could market on that basis, but some of the diagrams and pictures that I didn't quite get to the other day was how do you ever migrate from a tricycle to a bicycle because a bicycle is very unnatural and very hard to learn compared to a tricycle, and yet in society it has superseded all the tricycles for people over five years old. So the whole idea of high-performance knowledge work is yet to come up and be in the domain. It's still the orientation of automating what you used to do instead of moving to a whole new domain in which you are obviously going to learn quite a few new skills.
Last year, Tim Cook showed the staggering success of iPad sales in comparison to the PC industry:
Andressen Horowitz recently presented some interesting analysis on how 'Mobile Is Eating the World':
Which all jibes with market research firm IDC's prediction that 2 billion smartphones and tablets will be sold in 2017, in contrast to 300 million desktop and laptop PCs.
It’s easy to feel that technology marches forward with its own agenda, as if following some predetermined evolutionary pattern. But technology is nothing but an expression of man’s intellect and will. I would argue that at any point in the history of civilization, technology expresses deep things about a given society.
For example, clipper ships of the mid-19th Century perhaps expressed that era’s desire to make the world an interconnected and infrastructurally smaller place, in which long distance trade and travel could become routine.
Ours perhaps indicates a will to map all possible information and comprehensively understand the world, but more than anything indicates a will to communicate and socialize beyond the bounds of time or space.
And just like civilizations before, technology is held with a kind of religious reverence.
Ian Bogost argues that we’ve begun worshiping algorithms and their mysterious ways:
Here’s an exercise: The next time you hear someone talking about algorithms, replace the term with “God” and ask yourself if the meaning changes. Our supposedly algorithmic culture is not a material phenomenon so much as a devotional one, a supplication made to the computers people have allowed to replace gods in their minds, even as they simultaneously claim that science has made us impervious to religion.
This is an unsettling idea.
As my early investigations began to focus on a subject as large as our current and future relationship with technology, I felt I should start with a solid grounding in the past.
In 1974 Ted Nelson self-published an eccentric book describing computers as Dream Machines. These dreams would materialize in a new kind of media.
In the book he emphasizes the importance of our relationship with media, echoing Marshall McLuhan:
We live in media, as fish live in water.
He goes on to sum up the grand responsibility of designing it:
But today, at this moment, we can and must design the media, design the molecules of our new water, and I believe the details of this design matter very deeply. They will be with us for a very long time, perhaps as long as man has left.
My impulse is that we should keep this responsibility dear to us as designers, as we navigate uncharted territories. We’ve never had so many people using computers (and this new media) as there will be in the next minute.
What sets computers apart from all other tools previously invented is something called universality. The computer was initially built and understood as an "arithmetic organ", yet it turns out – as nearly everything can be translated into numbers of some sort – to be able to process just about everything: images, sound, text, you name it. Furthermore, as Alan Turing established in a shocking 1936 paper, certain computing machines exist called "universal machines," which can, by adjusting their configuration, be made to do absolutely anything that any other computing machines can do. All modern computers are such universal computers.
The first endeavors of my thesis work led me not down the path of exploring digital literacy, but digital creativity. This, before anything else, felt like the thing our transition to mobile could mean compromising on. As tools for sending and receiving information, and communicating, phones are surely the best tools we’ve ever had. But what about making new things? The rise of PCs democratized creativity like never before – we could have music, video, design and animation studios in one portable machine. But touchscreens don’t lend themselves so well to serious creativity. As makers, we use our hands. We need to feel. To develop muscle memory. Nothing we do meaningfully in the real world involves drawing with our fingers on glass, as Bret Victor nicely illustrates.
If mobile eats the world, and PCs become secondary or specialized devices, what would this mean for creativity?
At this stage I sought to understand more about input devices and, more fundamentally, some of the key tensions within the field of human-computer interaction.
Perhaps the most potent historical moment for me, through this lens, is the work of Doug Englebart. Deeply inspired by Vannevar Bush's essay ‘As We May Think’, Englebart pioneered both the mouse and graphical user interface that are so familiar today. He understood that the new 'universal computing machines' in development could be personal, and in being personal could augment a person's intellect and abilities. I love his use of the word augmentation – the lab he founded at Stanford was called the Augmentation Research Center – and he explained the concept as follows in this key report:
By "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. We refer to a way of life in an integrated domain where hunches, cut-and-try, intangibles, and the human "feel for a situation" usefully co-exist with powerful concepts, streamlined terminology and notation, sophisticated methods, and high-powered electronic aids.
I spent a few weeks towards the end of 2014 toying with possible physical input devices for creative work, imagining devices that could be paired with future mobile devices to compensate for the limited affordances of touchscreens.
Here are some of my prototypes:
But while I worked away at this, a sense that there was something deeper amiss in my anxiety about mobile devices loomed.
In a post at the start of the year I drew together some quotes:
Licklider wanted "...to enable men and computers to cooperate in making decisions and controlling complex situations without inflexible dependence on predetermined programs."
Maeda warns that "The true skill of a digital designer is the practiced art of computer programming, or computation."
Haverbeke determines "For open-ended interfaces, such as instructing the computer to perform arbitrary tasks, we’ve had more luck with an approach that makes use of our talent for language: teaching the machine a language."
The trend towards slick but obstinately inflexible, user-friendly but simplified, closed or otherwise gated systems and services has abstracted most notions of 'computation' so fundamentally that computers – and particularly the newer classes of mobile or smart device – have truly become 'appliances'.
Perhaps this was the transition that troubled me.
Consider this illuminating article preceding the iPad's release, which laid out pioneering Macintosh designer Jef Raskin's vision that 'an information appliance would be a computing device with one single purpose — like a toaster makes toast, and a microwave oven heats up food. This gadget would be so easy to use that anyone would be able to grab it, and start playing with it right away, without any training whatsoever. It would have the right number of buttons, in the right position, with the right software.' One single traditional computing device couldn't achieve this simplicity, but if the screen could morph to dynamically display different buttons it could...
600 million iPhones and 300 million iPads later it seems like that idea has legs.
So what might a transition to digital ‘appliances’ mean for our relationship with technology?
One of the most inspiring books I read in pursuit of my thesis was Seymour Papert's Mindstorms.
In the book, from 1980, Papert outlines his philosophy of using computation to help children learn, with examples from his research. Instead of poorer young math students being told they were simply wrong, they were to explore math problems through trial and error. This opens up new ways of critical thinking instead of closing them down. Any programmer knows the pain and joy of problem-solving. The traditional view that there is a right and wrong answer in math (in turn ensuring that poor-performing children assume they just don't get it) is painfully narrow-minded – as it would be to say a programmer is poor for not nailing a problem first time.
"The child programs the computer. And in teaching the computer how to think, children embark on an exploration about how they themselves think. The experience can be heady: Thinking about thinking turns the child into an epistemologist, an experience not even shared by most adults."
It's a wonderful book.
I wish I’d been exposed to LOGO as a child. It took me 29 years to discover my love for programming. I wrote last year about the joys of the experience.
Thankfully kids today are being nudged towards programming at an urgent rate, as the US and other countries scramble to ensure a competitive future in an increasingly technical global economy. There are code clubs, apps like Hopscotch and Scratch, online courses. It’s not an embedded part of the curriculum yet but it’s fast getting there.
This is great. But it will be two decades before the kids of today enter the working world. This will coincide with digital native millennials entering positions of leadership and power, calling shots in what will surely be a radically more technology-driven world than we have today (imagine).
The responsibility to shape that world falls upon all of us.
Consider this survey on the issue of Net Neutrality from last year:
The vast majority felt ill-informed to answer the survey, on a critical issue dictating the future of the web.
I began thinking about our understanding of technology as a form of literacy.
In Changing Minds, Andrea diSessa lays out a three-part framework for thinking about literacy:
First, there is the material pillar. That is, literacy involves external, materially based signs, symbols, depictions or representations. This last set of terms, as well as others, holds an essential magic of literacy: we can install some aspect of our thinking in stable, manipulable, and transportable physical form. These external forms become in a very real sense part of our thinking, remembering and communicating.
The second pillar of literacy is mental or cognitive. Clearly the material basis of literacy stands only in conjunction with what we think and do with our minds in the presence of inscriptions. A book is only a poor stepping stool to a nonreader. Material intelligence does not reside in either the mind or the materials alone. Indeed, the coupling of external and internal activity is intricate and critical.
The third pillar of literacy is social, the basis in community for enhanced literacies. Although one may imagine that an individual could benefit from a new or different material intelligence, literacy … is unambiguously and deeply social.
The computational literacy advocated by diSessa parallels Papert’s ideas. He imagines a world in which an ability to think computationally, use software to augment problem-solving, and generate new interactive forms is an everyday literacy – one embedded in society at an infrastructural level like math today.
At this stage in my research I came across the idea of ‘procedural literacy’ which tallies, and in many ways is interchangeable, with diSessa’s notion of ‘computational literacy’.
In Procedural Literacy: Educating the New Media Practitioner, programmer and educator Michael Matteas (who runs the Center for Games and Playable Media at UC Santa Cruz) argues that rather than get caught up in teaching specific programming languages to new media students, the important thing is to get them to think procedurally in the manner encouraged by the explicit logic of code. Furthermore, a grasp of such ways of thinking is important for the 'wider populace' to be more literate in the technology surrounding them.
In the article, Matteas describes a 1961 discussion between Alan Perlis, Peter Elias and J. C. R. Licklider. Perlis wants all students of the time to learn to think programmatically – not a specific language or machine but 'how analysis of some intuitively performed human tasks leads to mechanical algorithms accomplishable by a machine'.
In response Elias imagines that in the near future both computer programs and users will have become advanced enough that only a fraction need ever worry about programming itself. The rest may simply ride their wave.
Elias desires the development of frictionless tools that, like the computers on Star Trek, allow us to make the computer do our bidding with little work (e.g. “Computer, gather data on the anomaly, correlate it with all known space phenomena, and suggest a course of action.” Computer: “Done”). The problem with this vision is that programming is really about describing processes, describing complex flows of cause and effect, and given that it takes work to describe processes, programming will always involve work, never achieving this frictionless ideal. Any tools that reduce the friction for a certain class of programs, will dramatically increase the friction for other classes of programs. Thus, programming tools for artists, such as Flash, make a certain style of interactive animation easy to produce, while making other classes of programs difficult to impossible to produce. Every tool carries with it a specific worldview, opening one space of possibilities while closing off others. A procedurally literate practitioner will still make use of specific tools for specific projects, but will be aware of the constraints of specific tools, will be capable of considering a space of computational possibility larger than any specific tool.
Licklider's response to Elias is perfect: “Pete [Elias], I think the first apes who tried to talk with one another decided that learning language was a dreadful bore. They hoped that a few apes would work the thing out so the rest could avoid the bother. But some people write poetry in the language we speak. Perhaps better poetry will be written in the language of digital computers of the future than has ever been written in English.”
From this vantage point Matteas outlines some of his ideas for giving his students a firm grounding in procedural literacy.
Procedural literacy is not just the craft skill of programming, but includes knowing how to read and analyze computational artifacts.
The sense of reading and analyzing computational artifacts is critical. It's this that I think extends beyond any particular study field or domain – this understanding is necessary for everyone if they are to participate in a culture of pervasive technology.
Without a deep understanding of the relationship between what lies on and beneath the screen, scholars are unable to deeply read new media work, while practitioners, living in the prison-house of “art friendly” tools, are unable to tap the true representational power of computation as a medium. The ideal of procedural literacy as necessary, not just for new media practitioners, but as requirement and right of an educated populace, has been with us for over 40 years. Yet the two culture divide persists, with artists and humanists often believing that programming is a narrow technical specialty removed from culture, and computer scientists and engineers often happy to pursue just such an unexamined and narrow focus.
Hinging the issue on the 'Two Cultures' problem is a powerful point. While people may have argued about the division since C. P. Snow's lecture in 1959 (two years before the discussion above) the way technology and code has pervaded culture makes it more critical than ever.
In the same way that people engage language throughout their entire educational trajectory, so to should students engage procedurality. Only then will computation truly become an expression of culture on par with language, image, sound, and physical objects, adding process-based representation to the human conversation.
It’s bold and ambitious view that spoke to me deeply, providing a useful anchor for my explorations from here on.
Two other resources helped me focus on this theme, and shaped my understanding of how computational literacy and thinking were different from understanding how to use computers, and actually write code. These were: Understanding Computer Programming as a Literacy by Annettee Vee and Computational Thinking by Jeanette M. Wing.
I now had a mission statement:
Encourage and inspire a cultural shift towards two-way computational literacy, creating a more inclusive and participatory technology culture.
I also determined a set of benefits to computational literacy, the rewards for pursuing such learning:
Systems can't be controlled, but they can be designed and redesigned. We can't surge forward with certainty into a world of no surprises, but we can expect surprises and learn from them and even profit from them. We can't impose our will on a system. We can listen to what the system tells us and discover how its properties and our values can work together to bring forth something much better than could ever be produced by our will alone.
At the heart of my mission was, undoubtedly, an education agenda.
diSessa has a useful model for learning: the transition from functional understanding to structural understanding.
Functional understanding is, in simple terms, understanding how to use something. I may know which button to press to use a toaster, but not how the heating element or controls actually work – and therefore I cannot fix a toaster, or build a new one. Children learn intuitively about balancing mass on a seesaw, but they don’t necessarily understand force or torque. To transition to a structural understanding, a person needs to understand an underlying system. One example offered by diSessa is knowing how to maneuver a car during a skid. We could learn through instruction, or trial and error, to turn into the skid, but only with an understanding of the accompanying physics do we know why to do this – and only with that structural understanding could we confidently apply it to a new phenomenon.
This model has obvious application to computing. If software is understood through rote learning, or the limited grasp that ‘button X is meant to do Y’, the user may struggle if something goes wrong. Their ability to draw conclusions from errors, or make informed assumptions about new software, may be severely limited.
At the extreme end, this is probably why people anthropomorphize computers, and project blame when software ‘just doesn’t want to’ perform a function. But even in more advanced computer users, a lack of structural understanding isn’t uncommon. This isn’t the user’s fault of course – decades ago we settled with the idea that functional understanding (computer literacy rather than computational literacy) was perfectly sufficient for most people.
My aim has never tended towards introducing users to any specific programming language as such. In my examination of what a greater computational literacy might constitute, it became clear to me that an understanding of how computers work and what one can or cannot instruct a computer to do should come before anything else. Despite the greatest of intentions, this often feels missed from many recent 'learn to code' initiatives.
As outlined in this excellent article we transitioned from a pre-literate society in which an elite of scribes held a privileged position to the widespread common literacy we enjoy today. But we are still at 'scribe stage' with computational literacy – in which an elite hold a privileged position, and others remain dependent.
I opted to focus on adults as my target users for a simple reason: while we're finally making progress with bringing programming into schools, technology culture is moving so fast that a number of lost generations (including my own) risk falling behind.
The gap I had identified lay somewhere in the space between the idea of picking up programming having never occurred to a person, and their decision to learn. There is a wealth of tools and resources available for learning code, from online programs like Treehouse and Code School to adult learning initiatives like General Assembly and Flatiron School. My aim would be to bridge the gap that many people would never think to cross, with the benefits of programmatic thinking perhaps being unapparent or unaddressed. In that sense it would be an intervention and gentle persuasion piece.
Focusing on this older audience presents some interesting challenges for education. The older we get the less receptive to learning we frequently become.
When you are young, your brain is able to strengthen certain connections and weaken certain connections to make new memories.
It’s well known that learning languages is easier the younger we are, and unsurprising that this correlates to other types of learning too.
Worse yet, we we have a culture in which numeracy isn’t as valued as other forms of literacy. In his book, Innumeracy, John Allen Paulos observes that ‘the same people who cringe when words such as “imply” and “infer” are confused react without a trace of embarrassment to even the most egregious of numerical solecisms … unlike other failings which are hidden, mathematical illiteracy is often flaunted: “I can’t even balance my checkbook.” “I’m a people person, not a numbers person.” Or “I always hated math.”’
I’d argue that looking at code creates an equal, if not greater, anxiety than looking at algebra for most people.
It would be an uphill struggle.
Psychologist Piaget has a concept of 'assimilation' in his education theory, in which he proposes educators 'use familiar examples to facilitate learning more complex ideas'. This reasonable premise led me to explore ways I could start out in a familiar space, and gently introduce computational concepts.
Inspiration wasn’t far away, I saw it each morning on the subway.
It intrigued me that large swathes of the population who might not consider themselves particularly numerate or technical enjoyed the mental challenges of puzzle games. There seemed to be such an affinity between the type of logical thinking required by games such as Sudoku and Monument Valley as the kind of rigorous, mechanistic thinking required of programming.
In ‘What Video Games Have to Teach Us About Learning and Literacy’, James Paul Gee describes the ‘pleasantly frustrating’ feeling of playing games. This sounded a lot like my experience of programming. In fact, the trajectory of successfully learning something is often like that of a video game. You begin with a basic grasp, and work your way through increasingly complex challenges. The curve guides you through moments of frustration, to moments of relief. If you were dropped into a game at a later stage you would likely give up in despair, but had you followed a learning curve from the start you’d be prepared by this stage. A great analogy for learning in general.
I began focusing on the potential of games and play as a vehicle for my educational agenda.
Games occupy a fascinating space between freedom and rules. Without a certain tension between the two, a game cannot really be called a game. There need to be both constraints, and the potential to express free will and choice.
While I perhaps had an intuitive sense of this, I had never distilled it to such reductionist simplicity. It was compelling to frame games and play this way, and think about the opportunities they present for exploratory behavior and learning. Moving freely within delimited boundaries seems like a perfect environment for guided exploration.
There is a wealth of research supporting the need for play in childhood development. From developing language and imagination to creativity and social skills. Equally as interesting is the case for play beyond childhood. George Bernard Shaw joked that “we don't stop playing because we grow old; we grow old because we stop playing.” Research doesn’t disagree.
We, as homo sapiens, are fundamentally equipped for and need to play actively throughout our lifespan by nature’s design. While most social mammals have a life cycle that involves dominance and submissiveness (as in chimpanzee troops or wolf packs) with play diminishing significantly as adulthood arrives, we retain the biology associated with youthfulness despite still dying of old age! By this I mean that our overall long period of childhood dependency, which is dominated by the need for play, does not end with our reaching adulthood. Our adult biology remains unique among all creatures, and our capacity for flexibility, novelty and exploration persists. If we suppress this natural design, the consequences are dire. The play-less adult becomes stereotyped, inflexible, humorless, lives without irony, loses the capacity for optimism, and generally is quicker to react to stress with violence or depression than the adult whose play life persists. In a world of major continuous change (and we are certainly facing big changes economically now) playful humans who can roll with the punches and innovate through their play-inspired imaginations will better survive. Our playful natures have arrived at this place through the trial and error of millions of years of evolution, and we need to honor our design to play.
Video game theorist and advocate Ian Bogost has been a reassuring influence for my work. His position, in recognizing and singing the praises of games as a potent vehicle for learning (and, he might add, persuasion) has been an important reference point.
For years now, I have been arguing that games have a unique power to explain complexity. Unlike television and blogs and TED talks and even many long-form books and articles these days, games are the one popular medium that embraces complexity rather than shying away from it. Games endorse and even require systems thinking, the process of understanding the world as a complex network of interconnected parts.
Encountering his views, I was immediately drawn back to diSessa’s notion of structural understanding. If games ‘endorse and even require systems thinking’ then surely they hold unique potential for initiating players in certain kinds of structural thinking. If a game could nudge people meaningfully – and entertainingly – towards this kind of understanding, surely that would justify it as a vehicle for learning over more traditional educational models or frameworks.
Beyond this, I got thinking about the relationship between games and computing in general. With Bogost advocating the power of video games for developing what he calls ‘procedural rhetoric’ in his book Persuasive Games, I began drawing correlation to procedural thinking and literacy. He qualifies the term as ‘the practice of authoring arguments through processes’ and proposes that this affords ‘a new and promising way to make claims about how things work.’ Games are systems, with rules, that operate in time and some form of space. Playing a game means engaging with some form of procedure, which seems to eminently parallel programmatic modeling and thinking. My interest and intentions diverge with Bogost in that I’m not interested in persuasion or rhetoric as such, beyond convincing players that programmatic thinking is a valuable skill and practice. Predominantly my agenda lies in exposing players to a system in an effort to deliver some form of transferable structural understanding. Is it possible to build and expose a system without including an opinion or bias of some form? I’m not sure. There are compelling arguments that programming languages themselves shape culture, so I should no doubt accept that any system I design will contain such inherent biases.
As I pondered this, I came across the following quote:
The rise of computers has paralleled the resurgence of games in our culture. This is no accident. Games like Chess, Go, and Parcheesi are much like digital computers, machines for creating and storing numerical states. In this sense, computers didn’t create games; games created computers.
Eric Zimmerman – Manifesto for Ludic Century
This is a really interesting viewpoint though, as my advisor points out, the mechanics of those games depend entirely on the players and their (human) strategies.
Mechanics would clearly be of deep importance to any game I designed. There’s a reason there are game genres, and many variations on the same themes. Conceiving new games is hard. There’s a balance between ease and complexity, between strategy and luck, and between uniqueness and repetition.
As far as such balances relate to videogames, Atari founder Nolan Bushnell coined an interesting aphorism back in the 70s:
All the best games are easy to learn and difficult to master. They should reward the first quarter and the hundredth.
Game experts might disagree but I (crudely) see two broad types of game – those with a narrative structure and those without. The value and mastery of Chess or Tetris comes in repetitively trying to achieve a better result in a constrained domain. There may be a start, middle and end, but at any moment you are working in much the same game space. On the other hand, a Mario or Grand Theft Auto style game follows some form of narrative. You explore and progress through different spaces and challenges, and the mechanics shift slightly as you do so.
Casual puzzle games can fall either side of this. In Dots the mechanics don’t change greatly, but you progress through different levels or challenges. Even something like Sudoku arguably follows this pattern. In Threes, however, you repeatedly play in the same game space, with identical mechanics and no narrative. Your goal is simply to better your last result. Chess and Go aren’t so different, good players want to win faster, better, more elegantly.
In terms of video games, these types present very different design challenges. Should I take people on a journey, or build a robust stable game space that rewards repetition and mastery? And should the game be played alone, or with one or more others?
My thesis project wouldn’t technically be the first game I had designed. But that limited experience placed an emphasis almost exclusively on creating compelling and innovative content, not on defining novel mechanics. The mechanics were familiar; tried and tested.
In developing a game concept, I needed to hit a set of goals (and find a delicate balance between them):
I set myself the following direction:
Positioned as ‘pick up and play’ game rather than an educational app, the product will introduce a narrow range of programming concepts. For each turn, the player will need to construct some basic logic. Through this activity, methods of ‘instructing’ a computer to do things are exposed. Play should follow a learning curve, demand strategic thinking, but remain lightweight and fun. The takeaway, if successful, will be a small but inspiring insight into programmatic thinking.
To the first point, I felt it was important not to present the game as overtly educational. The reason being that if the app was presented as about ‘learning programming’ it would immediately limit the audience to a self-selecting group that on some level wanted to learn programming in the first place. I want to reach people to whom this has perhaps never occurred, but who might just find they enjoy the requisite thinking. For this reason, casual puzzle game enthusiasts seemed like a good demographic to target.
As touched upon above, games necessarily contain systems. The opportunity that I wanted to seize was in exposing and allowing this system to be explored in such a way that something about programming and programmatic thinking was conveyed. This where the idea of ‘instruction’ rather than direct manipulation being a key game mechanic came from.
My notebook is full of tiny sketches that I would hate to force anyone to make head or tail of.
Luckily I tend to shift quickly to high-fidelity mockups and exploration.
Shortly before Spring Break at the start of March I had a rough concept plotted out. I designed and produced a paper prototype which I got classmates to test, but unfortunately the nature of the game didn’t really lend itself to paper. Dynamic media doesn’t always transition well to static, surprisingly enough.
The concept was simple – a classic black vs white 'defeat the opponent' game in (loosely) the vain of Go, Checkers, Chess or Othello. The twist was that you would never directly move the game pieces by hand, they would be manipulated through some basic programming logic. My hope was that combining this control method with competitive strategizing would help deliver and reinforce a grasp of the following programming concepts: conditional statements, Boolean operators and some basic positioning and scale ideas.
Pieces could be removed from the board in two manners: scaling down to 0, or scaling up enough to cause a collision, in which case all colliding pieces burst.
Much of the strategy rests on two principles – you can't target by color, so your conditional selections are likely to include your own pieces. You scale both, but aim to lose net less of your own. Additionally, there is always a tension between scaling up and scaling down. Sometimes scaling up is defensive, other times it's an offensive tactic (in pursuit of causing a collision and consequent burst).
I worked through a few bold, colorful designs. Early on I received feedback that making the game pieces look tangible, but then not allowing direct manipulation (i.e tapping and moving the pieces) could cause frustration. I stripped the game pieces back to minimal ring shapes.
Over the course of 4-5 days I built a fully functioning prototype with EaselJS. This was my first encounter with the library, which is designed for HTML5 Canvas games – seemingly ideal. I made as many features as possible flexible. For example, it's simple to change the size of the grid and number of pieces.
While the ideal for a two-player game of this nature would be asynchronous play, for now it's 'pass and play' – meaning both players share the same screen, and simply take turns.
I opted to leave the buttons and input features with no design polish at all, and I postponed looking at any animations (for which the accompanying TweenJS library looked useful). These things didn't seem important to the MVP.
By the middle of Spring Break I had a game. Here is the code.
Without wanting to dive too deeply into the code, there was one component that seemed compelling in relation to the way I was trying to get players to ‘instruct’ programmatic changes in the game space. Certain buttons that players would use stored snippets of code that very directly correlated to the role of the button in the game. There was a certain purity to this.
For example, consider these conditions buttons, and the code they stored and passed to a larger function when taking a turn:
BIGGER THAN:
function(a, b) {
return a > b
}SMALLER THAN:
function(a, b) {
return a < b
}EQUAL TO:
function(a, b) {
return a == b
}NOT EQUAL TO:
function(a, b) {
return a != b
}Or these Boolean operators:
AND:
function(rule1, rule2) {
return rule1 && rule2
}OR:
function(rule1, rule2) {
return rule1 || rule2
}It was not too much of a stretch to say that players were engaged quite directly with a form of ‘programming’.
I played the game with others multiple times, and gathered plenty of valuable feedback. When I conceived the game I'd imagined the goal to be removing your own pieces first, but it quickly became clear that removing the opponent's pieces felt better (and more in the vein of traditional games, of course!). This was a trivial change, with no other impact on strategy.
Next was a lot confusion with the AND / OR logical operators. This wasn't unexpected, and in many ways is an objective for the game – making something unfamiliar familiar. The prototype doesn't have an onboarding process, making it hard to judge the scale of the issue. But it's clear that not understanding the programming principles leads to frustrating gameplay – something to be acutely aware of whichever direction the concept goes in.
There was smaller feedback on the uncertainty created by highlighting selected rings in green. Once highlighted, the original color isn't visible, so you're not sure how many of your own pieces vs your opponent's you're targeting. This can easily be addressed in design.
People also wanted to see whose turn it was, and a count of the remaining pieces. Both things I'd intended to include, but left out of the prototype.
Lastly, and most significantly: once people had grasped the logical approach to conditionally selecting pieces (i.e building the logic of their turn) it became laborious and repetitive. I noticed that people started leaning on the logic they had grasped, rarely trying to get more ambitious with longer statements (e.g using three-part conditions with two operators). Regardless, this feels like a fundamental problem – and it cuts to the heart of what I'm trying to achieve. What is the balance between learning and playing? Once people have learned the concepts I'm hoping to convey, should it just be a fun game, or should they be moving on to something more challenging? The tension between education and entertainment is delicate, and critical.
I was left feeling that the game needed considerably more to it – both strategically in the game mechanics and educationally in the programming principles included.
In class we presented the status of our projects, and received jars full of feedback. Some of the key questions people thought I should be asking were:
Following valuable feedback on the 'Circles' game I went back to the drawing board, seeking to take the things that were working but build something with more depth, fun and replay value.
One of the biggest flaws with the first game was that once people had grasped the way the conditions and logic worked, and could see what they wanted to do on the board, getting there felt like work. Also, the unpredictable nature of exploding circles made it less compelling to delicately target specific pieces on the board.
Amongst some wilder ideas, I put this rough mockup together:
This attempted to build on the first game by introducing some further vectors. Instead of targeting circles by position or size, players would target by position, shape or opacity. Instead of growing or shrinking circles, players would transform shapes or change their opacity. The goal would be to capture shapes by increasing their opacity to 100, but the catch was that you could play defensively and decrease the opacity of your opponent's pieces. Furthermore, you would build sequences of four steps for each turn allowing more sophisticated strategy.
It had the kernel of a new direction, but was a little convoluted.
This morphed into a next mockup:
This lost the opacity vector, and simplified the shape vector to being about transforming between circles and squares in two-parts.
Next came four-part shapes:
This was starting to get interesting. Players would seek to "capture" shapes by completing circles or squares in their respective colors. The sequencing opened up interesting possibilities, where the state of shapes would change between steps allowing players to do smart things.
I liked this because it felt like it aligned with the kind of programmatic thinking I want to encourage and introduce people to. Much of programming involves holding a piece of a system in your mind, then thinking a few steps ahead about how it's going to change – and what consequences that may have. For example, a programmer may need to anticipate the value of variable x at the end of 10 iterations of a 'for loop'.
I then decided that a goal to create shapes of either type (i.e circle or square) for each player was a little confusing, perhaps one player could chase complete circles, and one squares?
Going back to black and white basics, I mocked this up:
Now player one would try to capture white circles, player two black squares. The sequence element was in place. Actions would also include rotation (which wasn't possible with the two-part shapes). It began to feel like a good level of complexity to be challenging and strategically interesting, and I was hopeful it continued to express my goal of conveying programmatic thinking.
But would it be fun?
It was time to test.
I built an interactive version of the game board in JavaScript, as managing the morphing shapes in paper wasn't really workable. Then I printed the other game components and instructed some other IxD students to play. Based on the sequences they built, I would manually update the game board on screen.
The tough thing with a paper prototype is that there are no explorable affordances. You can't pick up a piece and see what you can do with it, or try tapping things. This made the process a little contrived, as I had to explain the game more than I wanted.
Once the rules were grasped the students seemed to really engage with the cerebral challenge of thinking ahead in the sequence, so it felt like a promising start.
I decided to move forward and build a prototype.
For a week or so I began spending 10+ hours a day coding the game. This was an intensely challenging but rewarding period. It felt like a sketch that I was gradually coloring in and detailing. Interestingly, I wasn’t so much recreating designs as doing design work in code. I’m not quite proficient enough to always work like this, but I could see real value in simultaneous design and prototyping in my future work.
I pushed updates to Github over 40 times over the last 10 days of March.
It wasn’t yet perfect, but it wasn’t meant to be. I had enough to continue testing.
I had 4-5 students from other SVA programs play the game as a fully functioning web app, during a social event. It's fair to say the depth of thinking required for the game and a bustling informal setting aren't always compatible, but I observed that once people engaged focus they found the game really challenging and interesting. At least one person got stuck in, and asked to be informed when the game is released.
The biggest takeaway was that onboarding is critical. It took significant explanation from me before people ever crossed the line from perplexed to excited. I won't be there to hold hands in the real world, so the tutorial / onboarding needs to be excellent.
Another key decision to make was the end or winning state for the game. The mechanic of selecting and transforming shapes was working, but I had to decide whether I wanted a winning player to have converted every shape on the board, a majority (9 or more), or take a different angle and try to connect four shapes in a row or column. I also had a fundamental decision to make in whether or not to ‘lock’ shapes once they were completed circles or squares. This would be rewarding for the player, but not doing so could provide an interesting further game dynamic.
I began leaning towards a) the “connect four” end state and b) not locking shapes. Connecting four could be a quicker and potentially less intimidating game than the other options. Not locking the completed shapes introduced some interesting new strategic elements, particularly because you can never select a circle or square directly through a circle or square condition piece.
A week after the first prototype test I attended NYU Game Center's Playtest Thursday – a weekly event where anyone can bring along a prototype game for feedback, which is a great resource for the community.
This was to be the first time people with no connection to me or the IxD program would see and play the game, which was a little nerve-racking.
Since the last round of testing I'd built an interactive tutorial, and was hopeful people would pick up the game via the tutorial rather than any support from me.
In total, six people tested the game. People were enthusiastic, but struggled with the aforementioned onboarding – it was nowhere near solid enough yet.
Specific feedback I captured:
One of the testers taught programming, and followed up the next day to say he felt the game aligned well with his interests and he’d be happy to be involved. We discussed some of his feedback over email, particularly around the first player advantage issue.
At this stage the two dominant concerns were onboarding and first player advantage.
Here is a walkthrough of the tutorial as it stood during the previous day’s testing:
I decided to rework the tutorial in two fundamental ways: first, splitting out learning about conditions and actions into two steps and second, taking people into the game itself without further grounding and explanation. The latter felt particularly important as people had expressed a disconnect between the tutorial and entering the game. They couldn’t see a clear enough relationship between the two.
I rebuilt it with an interactive demonstration of the switch actions first, then the conditions and logic. Following that players would enter the game but before playing see an explanation of each component (the grid, then the conditions, sequence and actions panels).
The first player advantage was undoubtedly a problem. In my own testing, I had on occasion been able to win the game on the first turn. Which, of course, isn’t much of a game. It underlined how much could be achieved with a four-step sequence, which is also a positive – when you play a strong turn and watch it unfold it’s very rewarding.
I contemplated a few solutions. Simply having a shorter sequence was one, but compromised the potential for sequential thinking. Giving player two black segments on the opening game board was another, but felt a little awkward and wasn’t clearly going to be fair. Restricting the available actions or conditions for player one was another possibility. In the end I opted for something else: restrict the sequence to one step for the first turn, two steps for the second, up to four steps for the fourth and beyond. This would apply to both players. The positive repercussion for this was a gentler ramp into the game. Testers had expressed being a little overwhelmed with choice at the start of a game. This would reduce the burden – only one move to consider for the first turn, building to two and then three and then four as the game progressed.
I built this into the game, communicating the constraint with a padlock icon and “Locked until next turn” text.
Around this time I was also tackling a major technical challenge. I wasn’t sure if it would be feasible, but I wanted to try to get the game working natively on iPad – meaning it could be downloaded from the store rather than played as a web app. A first possible solution, using a tool to convert a CreateJS project to native code proved to be a non-starter. It was out of date and I couldn’t get even the demo projects to run on iOS8. Instead, I started looking at PhoneGap and its counterpart Cordova. Neither would be a perfect solution, in that running a web app inside a native shell would never be as performant as native code. But it had potential.
I began experimenting, and got a version running. It was slow, and buggy. But it was running.
Over the coming weeks I managed to get it into much better shape. I found a plugin to run the web app in WKWebView not UIWebView which would harness Apple’s Nitro JavaScript Engine. I fixed a few API bugs, and I refactored a lot of the game code to make it more efficient, compensating for some of the performance loss.
On April 13th I submitted the app to Apple for review.
Meanwhile I continued testing and refining the game. I had my roommate, a musician, produce some music for the game (three loops: one for the start screen and tutorial, one for the gameplay, and one for the winning screen). I also had it mastered, optimizing it for iPad speakers. I received some useful late feedback on the onboarding, being that the value of the flip and rotate actions was getting a little lost. Firstly these aren’t directly covered in the tutorial, and second they have little use in the first turn (it will never be worth simply flipping or rotating shapes without also switching corners, being that the first turn only has one sequence step). I compensated, hopefully, for both by locking the flipping and rotating actions in the first turn, then extending the tutorial to explain them as they become unlocked on the second turn (if the player had entered the game via the tutorial, else they would simply gain access to these pieces on the second turn without requiring explanation).
It was feeling pretty solid.
This is the final JavaScript code for the game, all 4615 lines of it.
To avoid falling to the back of the App Store review queue I sat on the newer version and awaited Apple’s decision. In the meantime I began working on this website. I wanted it to have three sections: a simple marketing site for the app, and space to detail more about what people would learn from the game (which would be linked from the app for those interested in pursuing further), and of course this documentation of the process.
I built the site to be responsive, using the Skeleton CSS framework.
The learning section felt really important. While I had opted to go light on the educational agenda of the game (occasionally imagining it as a kind of educational Trojan horse) I knew it would be important to enable those who engaged more fully with the learning side to have somewhere to go with their interest. A dead end would somewhat undermine the game.
The journey would be as follows: players could look at the ‘Next’ section within the game which broadly documents three programmatic structures introduced in the game: selection, sequence and loops. From here they can click to be redirected to the learning section of the site. At the end of this page a list of further resources is provided, including a Treehouse course on digital literacy.
On April 24th, after 11 days of waiting, the app was approved by Apple. Huge news – I would be able to provide a download link when presenting my thesis, and get the game into the hands of real users. I submitted the much-improved update, which should be live before the Thesis Festival. The prototype was done, and it was no longer really a prototype.
When I set out to join the Interaction Design program, the old adage about 'getting out what you put in' felt resonant (if obvious). This was a space, structured yet open-ended, and the onus was on each of us to make it a vehicle to take us where we wanted. Our theses, of course, would be the ultimate embodiment of that.
I began this academic year with a job secured and life awaiting me in California. I was determined that I'd use the time wisely, as a unique opportunity for me to work on specific parts of my practice.
Previous theses have taken all shapes and sizes. Sometimes they are research-heavy. Sometimes they are deeply speculative. Sometimes they're whimsical, other times serious.
My work and academic background has afforded me plenty of opportunities for thinking and strategizing, but not much for making (until very recently). I was determined that I would place emphasis on bringing something concrete into existence, something tangible and polished. It would be a year of making.
With this agenda, my thesis would be a vehicle for upping my game: my craft and practice.
The meta narrative for this project is that less than two years ago I was also afraid of code. I also thought coding 'just wasn't for me'. It was the single most intimidating aspect of the curriculum. But strangely I took to it, with a passion. It became my favorite part of the program, the part I ended up lamenting there being too little emphasis on. I went from 'digital native' to 'code immigrant', and built something directly on the back of that specific personal experience. Something that depended on and furthered my own learning. Something that compounded that initial revelation.
In building something that sought to share that potential revelation with others, you might say I tried to pay it forward.
I'm hyper-critical of myself, and always pondering the what ifs and shortcomings. This project has many. But I'm proud that I delivered on my overarching goal. I didn't produce a fiction, I produced a reality... one that at the time of writing (less than two weeks after launching) has seen 300+ downloads across the USA, Canada, Latin America, Caribbean, Europe, Asia Pacific, India, Africa and Middle East.
I couldn't have done it without you. Thanks, SVA IxD.
It’s been an intense year. I'm filled with gratitude, which is lucky as I owe it to a lot of people.
Lulu for her patience and support from afar. My parents for encouraging me to finally make the move to NYC, and helping make it possible. Matt for demonstrating something done late is better than never at all (joke!). Ted for his wisdom and course-correcting. Maria for her musical talents. Luke and Aastha for generating illusions. Martha for playtesting and helping me retain a semblance of a social life. Jeff who first quelled my fear of code. Eric who steering us inimitably. Liz for making any of this possible. The rest of the IxD faculty, in their infinite talents and enthusiasm. And of course, the class of 2015 – we did it, guys.