Final project: Sound Kites

Get the code: Arduino + Processing

My final project for Physical Computing (with Yu Ji) is called Sound Kites. It is an installation/space consisting of sheets of fabrics and materials hanging from the ceiling to a height of about seven feet off of the ground. Attached to these sheets are strings which trigger a soundscape when pulled on.

This project evolved from a few different ideas that we had which centered around using a fan or source of air to create interesting interaction. One idea was to amplify a user's breath by syncing it with a fan. Another idea involved using a fan to float the sheets of fabric in order to produce a buoyant effect. Some of this early work can be seen in one of Yu's blog posts.

After some deliberation, we removed the fan from our project entirely. It wasn't clear what the fan was adding to the experience, and we were able to create the interaction we desired without directly using air in this way.

The circuit is very simple - there are 3 stretch sensors and 3 pull-down resistors connected to an Arduino Uno. Pulling on a string stretches the stretch sensor, causing a change in resistance which is picked up by Processing (working with Arduino through serial). Two different thresholds for the sensor values cause sound to be played either normally or with a phaser effect.

The sounds were found on open-source audio websites like the Internet Archive, Freesound (windchimes), and Behance (i/dex). I edited the three sound files to make their keys more similar, and we only used portions of the files. The sound loops continuously and only the gain setting is affected when a string is pulled, meaning that it would be possible to synchronize sounds to give the user even more complex interaction.

We used the Minim audio library to play sounds through Processing, which caused us problems with the quality of the audio playback when changing the gain. This introduces pops and clicks into the audio which definitely diminished the quality of the interaction experience. If this projects goes forward we will need to explore other options for producing sound.

The project was well received by our class. One interesting thing was that most of the users, when asked later, didn't know where the sound was coming from (computer speakers sitting on a shelf). This pointed to the fact that we had completely engaged them to the point that they didn't even think about the system's operation. This fact was a big achievement for us and will undoubtedly influence our future design decisions for this project and others to come.

Final project plan

Description

My final project (with Yu Ji) is an interactive system that allows a primary user to amplify both their breath and image. The primary user will blow into a sensor controlling one or several fans, and the speed of the fan(s) will mimic the strength of the user’s breath. The primary user’s face will also be projected onto to a screen behind the fan, making the interaction more personal.

The other users (people in front of the screen/fan) will work together with the primary user to make musical tones by putting objects directly in the stream of air created by the fan. The exact interaction and physicality has not yet been decided.

Timeline

Tuesday, November 12, 2013 
Project plan and bill of materials

Tuesday, November 19, 2013
Prototype with all components (blow sensor, fan, screen, camera, and ?)

Tuesday, November 26, 2013
Finalized design and implementation with two-way interaction

Tuesday, December 3, 2013
Project done


Testing plan

1. Build prototype
2. Test prototype with friends/colleagues
3. Incrementally test while finalizing interaction design
4. Test final version

Bill of materials

Box fan	$18.88
Nicer piezos	? x $1.50
Borrowing a lot of stuff	$0.00
TOTAL:	$18.88 + ?

Midterm project: Color flute

Get the code: Arduino + Processing

My partner (Yu Ji) and I decided to expand on my earlier project and create a device similar to a flute which could be used to draw colors on a screen. From the start Yu had wanted to use a sensor to detect blowing air, and a woodwind instrument seemed to be a good metaphor from which to start the design process.

I drew a rough diagram of what we thought the device should look like, along with a few details on what sensors we wanted to use. We were going to use a MPX5100GP air pressure sensor but it proved to require a pressure far too powerful for a device meant for human breath input. Instead we used a simple piezo buzzer.  The rest of a diagram is accurate - three photoresistors, one for each RGB color channel, and an accelerometer to detect the device's movements.

While looking for materials to house the device, we happened upon a long bamboo pole.  Wanting to be economical, we decided to use it since it was free, but we also realized that it was aesthetically pleasing to combine a natural wood with bare sensors.  I cut a small length from the larger bamboo and drilled small holes for the photoresistors.

For testing purposes, we wired the necessary sensors to an Arduino Uno using a breadboard. It was at this stage that we decided to binarize the photoresistor values since it was difficult to achieve accurate results in dynamic lighting environments. Choosing a threshold value between "on"and "off" allowed us to use the photoresistors much like the holes on a woodwind instrument.

Once we got everything running fairly smoothly, we miniaturized the circuit on two tiny breadboards hooked up to an Arduino Micro. I set aside channels for power and ground, and tried to minimize the area of the circuit by grouping like components together while allowing for sufficient space for the necessary wire connections.

One of the hardest parts was inserting the miniature circuit into the bottom of the device.  We used wires which were too hard to bend, and as a result at least one wire would come unhooked each time we installed the circuit, forcing us to repeat the process a number of times until the serial monitor confirmed that all of our sensors were functioning as desired.

The finishing touches were acrylic glass end caps that I cut to hold the piezo in place at the top of the tube and the circuit in place at the bottom of the tube. I happened to have acrylic scraps that exactly matched the color of the photoresistors, which was a bonus.

Above is the final result. This project was a challenge and it really helped my understanding of the device design process, from rough sketches to prototyping to fabrication to circuit optimization. It also highlighted our respective strengths. I was able to contribute much of the code and some of the elements of fabrication, and the "optimized" circuit I designed would have been a disaster without Yu installing the sensors into the physical device, making sure to keep the bundles of wires separated and marked, allowing for ease of troubleshooting later.

Response: Revisiting the definition of physical interaction

In my first post about physical interaction, I defined it as "two-way communication...between an object's physicality and/or the physical senses" used "in meaningful ways to achieve some end." Looking back, I still think this definition makes sense, but I have gained more insight into the nature of the interfaces providing this interaction.

Tom's theory that "nothing is intuitive" is important to me because it eliminates absolutism from interaction design.  Good design pushes boundaries in ways that can be adapted to rather easily, and it is in the design that the "user manual" should be coded implicitly.

Given this new take on what we are capable of, both as designers and users, I will continue to feel more confident in proposing design decisions that may not be classically accepted, as long as the user can learn in a way that recalls past experiences.  I am also optimistic that we as a technology-producing society will be freer to experiment with nontraditional paradigms that will speed the rate of advance in interface design.

Drawing colored ellipses with a glowing box

My assignment (with Sheri Manson) was to make a physical interface for use with a Processing sketch from Introduction to Computational Media.  I waived this class but Sheri had a sketch for use with this project.

Her original sketch was a canvas on which you could draw multi-colored ellipses using a mouse. The color, opacity, and audio was determined by the mouse pointer's coordinates and the speed of motion. We modified this sketch to use an accelerometer as a controller, with changes in motion along its x-, y-, and z-axis controlling these same attributes.

We mounted the accelerometer (with Arduino) in a semi-transparent acrylic box, along with a multi-color LED that we programmed to match the color on the canvas.  The movements were very fluid; however, the LED proved to be unable to represent the entire color gamut of a computer monitor. Nonetheless, the glowing box created an interesting emotional feeling as its true contents remained hidden from view.

In-class: Musical instrument with ultrasonic distance sensor

Get the code: ping_scale

My partner and I used the PING))) Ultrasonic Distance Sensor to play a musical scale with square waves generated by Arduino's tone().  

The sensor provides a time value for how long a sound wave takes to return after bouncing off of an object, which can be used to determine the distance from that object.  In this case, the object was my own hand.

We physically limited the scale to 32" to keep things manageable (the sensor a maximum range far greater than that).  Dividing this distance into 8 discrete parts gave us the "keys" for a one-octave scale.

We mapped the parts to the notes, and when my hand was a certain distance from the sensor, the corresponding tone was played.

Response: Bad elevators

I observed the people using the left bank of elevators in the Tisch School of the Arts to determine what interactions take place and how they happen.  Like most elevators, they have buttons which illuminate when pressed to show which floors have been selected.  Unfortunately, there are two main problems with these buttons.  The first is that pressing the buttons is fairly unreliable, and the second is that the buttons' illumination is extremely faint.

For the frequent elevator user accustomed to accurate button inputs and clear signaling, the Tisch elevators can be challenging.  Pressing the button once, however firmly, does not guarantee anything, and you can forget about a casual glance at the button panel, unless you want to go somewhere other than where you want to go.

The experienced Tisch elevator user knows the ways of reducing the chances of accidentally skipping your desired floor.  They know that repeatedly jamming the button improves your odds of the elevator system registering your button press.  They also know that there is no possible way to tell what floors have been selected unless you get your eyes within an inch or two of the button panel.  For those who don't want to stoop, repeatedly jamming the button regardless of if that button has already been pressed usually does the trick.

The transaction time is a few seconds longer than it has to be, which is plenty annoying, but the real downside is that the current state of interaction creates a feeling of uncertainty for everyone in the elevator, and worse, causes problems for people with physical disabilities who often have to take the elevator.

Lab: First Arduino program

Get the code: _3_speed_blinking

Our first Arduino circuit consisted of a switch controlling two LEDs.  The default state was the red LED on and the yellow LED off.  Pressing the switch reversed this (yellow on, red off).

After getting this working, I decided to try something more complicated.  I programmed the LEDs to blink at three different rates and use the switch to change speeds, partially by adapting this Arduino blinking example for use with two LEDs at once.

Overall the setup works well, although there is a small bug in the program that is unnoticeable during normal use. Bonus points if you can find it.

Deformable display from paper, Kinect, and projector

Flexpad is a technology that allows for the manipulation of a display, in this case a sheet of deformable paper material, to directly affect the image on that display.  It uses the Microsoft Kinect to sense physical changes and translate them into a correspondingly deformed image.

An apparent difficulty in this prototype is keeping the image perfectly aligned on the sheet - the blank edge of the sheet is frequently visible as the system constantly adjusts the projection to fit. However, this technology might be more successfully applied to a stationary object, perhaps mounted on a table or wall.  The system's response to the manipulation of an object seems quite accurate, and it is only the moving of the object through space that seems to cause problems.

The obvious drawback of this technology is that it requires not only an external sensor (the Kinect) but an external projector as well.  This is why I think it would be so much more useful if the object to be manipulated was physically anchored in some way.

In-class: Understanding electricity

We spent most of today's class building basic circuits.  One of the goals for the class to create a zoetrope using a small motor.  My partner and I did not make it that far but we did make a few different basic circuits, including this one:

I had mounted my Arduino on the breadboard earlier but it was not used in today's circuits. The active components include:

• 12V power supply
• Voltage regulator (5V)
• Three switches (in parallel)
• LED

This is the corresponding circuit diagram:

Pressing at least one switch made the LED turn on.  This particular circuit was very helpful in my understanding of the differences between connections in parallel and in series.  A circuit like this can be thought of as an OR logic gate, where it takes just one switch to turn on the LED.  If the switches were instead wired in series, all three would need to be pressed at the same time in order to turn on the LED, as would be the case with an AND logic gate.

Response: Defining physical interaction

The Art of Interactive Design by Chris Crawford
"A Brief Rant on the Future of Interaction Design" by Bret Victor

Crawford and Victor both agree that two-way communication or signaling is a necessary aspect of interactivity. However, while Victor espouses the implicit physical interaction with a book or a glass of water, Crawford denies such objects' interactive capabilities.

Unfortunately, I think Crawford mistakes content for medium. A book's content is its words and images, and its medium is the physical presence of its paper, pages, and binding.  Both content and medium must be analyzed together as parts of a whole system in order to understand that system's level of interactivity.

It is true that a book's content does not change, but it can be argued that its physicality does change. As a book is read and manipulated, its physicality is altered depending on, among other things, the way the book is handled and the page currently being read. Similarly, a glass of water changes through the act of holding, drinking, and refilling.

I would define physical interaction as two-way communication or signaling based on an object's physicality and/or the physical senses of the user. An object could be said to be physically interactive if it can both accept and provide sensory information to and from a user. Good physical interaction, then, requires the manipulation of physicality and sense in meaningful ways to achieve some end.

Apple TV is an example of digital technology that is not very interactive by the above definition. The interaction is mediated through single button-presses that keeps the user sensorially disconnected from the technology.

Fall 2013 courses

Applications (Nancy Hechinger)

This introductory class is designed to allow students to engage in a critical dialogue with leaders drawn from the artistic, non-profit and commercial sectors of the new media field, and to learn the value of collaborative projects by undertaking group presentations in response to issues raised by the guest speakers. Interactive media projects and approaches to the design of new media applications are presented weekly; students are thus exposed to both commercial as well as mission-driven applications by the actual designers and creators of these innovative and experimental projects. By way of this process, all first year students, for the first and only time in their ITP experience, are together in one room at one time, and as a community, encounter, and respond to, the challenges posed by the invited guests. The course at once provides an overview of current developments in this emerging field, and asks students to consider many questions about the state of the art. For example, with the new technologies and applications making their way into almost every phase of the economy and rooting themselves in our day to day lives, what can we learn from both the failures and successes? What are the impacts on our society? What is ubiquitous computing, embedded computing, physical computing? How is cyberspace merging with physical space? Class participation, group presentations, and a final paper are required.


Physical Computing (Tom Igoe)

This course expands the students' palette for physical interaction design with computational media. We look away from the limitations of the mouse, keyboard and monitor interface of today's computers, and start instead with the expressive capabilities of the human body. We consider uses of the computer for more than just information retrieval and processing, and at locations other than the home or the office. The platform for the class is a microcontroller, a single-chip computer that can fit in your hand. The core technical concepts include digital, analog and serial input and output. Core interaction design concepts include user observation, affordances, and converting physical action into digital information. Students have weekly lab exercises to build skills with the microcontroller and related tools, and longer assignments in which they apply the principles from weekly labs in creative applications. Both individual work and group work is required.


Materials and Building Strategies (Peter Menderson)

You’ve built a foam prototype. Your project idea is now out in the open sitting on a table where you and your teammates can look at it. It’s not quite what you thought it would be when you made your first rough sketch, there’s even something a little goofy about it, but then there’s also that interesting curve that you hadn’t envisioned. Your teammates have also noticed some things that you hadn’t thought of. You see where you can reshape the foam to make the prototype both look and work better. You’ve made your first step; you’ve moved your project forward. Removing barriers to creative problem solving and learning the steps for advancing a project are the dual purposes of this course. You’re asked to make things over and over during your time at ITP. This class helps you to break out of 2-d screen and keyboard thinking and take advantage of the discoveries that inevitably occur when you're thinking in 3-d by manipulating materials with your hands, observing the results, and refining successive iterations of your idea. From techniques for prototyping and making small objects to fabrication methods for kiosks, you’ll get hands-on experience with a variety of materials and methods. You have an idea for a wearable device? Mock it up with the sewing machine. You're thinking about a squeezable children’s toy with sensors? Make a mold and cast some sensors inside soft rubber. You want to build an installation? Make a foam core model of the space and get a valuable preview of your project installed. During the course you'll be introduced to building in a variety of materials. You’ll make objects of wood, foam, plastic, metal, clay, plaster, rubber, paper and fabric. You’ll move a project from sketch to prototype to presentation and learn to incorporate the lessons of the process into your final product. By taking notice of the unexpected your original concept will evolve, and amplified by those revelations it will surprise you and delight your audience.


Designing for Persuasion (Katherine Dillon)

In subtle and not-so-subtle ways technology is influencing our behavior – from buying more books on Amazon than we intended to, to helping us change bad personal habits to leveraging the voices of many– technology presents an opportunity to be an agent of change. This 2 pt course will explore how technology can be used to influence behavior. We will look at a number of behavioral theories including incentive–based design, gamification and social influence. We will review case studies on how these techniques have been used to effectively affect behavior. After researching theories on behavior motivation each student will identify a problem or issue that they hope to influence. Students will document the problem, develop a concept to influence the behavior associated with that problem and prototype (or build) their solution. They will test their solution and draw conclusions from the experiment. Projects can attempt to influence social change at a large, social scale or at a personal level. The unifying theme behind the projects will be that they intend to inspire positive change.


Visual Language (Katherine Dillon)

The goal of this course is to provide students who are new to the principles of visual design with the practical knowledge, critical skills and confidence to effectively express their ideas in a visually pleasing and effective way. Over the course of 7-weeks an overview of the many tools and techniques available to convey an idea, communicate a message and influence an experience will be presented, discussed and applied. Topics covered in the course include: typography, color, composition, branding, logo and information design. This class is intended for students who do not have formal graphic design or visual arts training but recognize the powerful impact of visual decisions in their work.