Koji Yatani, Ph.D.
		I am an associate researcher in the Human-Computer Interaction group at Microsoft Research Asia in Beijing. I received my Ph.D. at University of Toronto under the supervison by Prof. Khai N. Truong at Dynamic Graphics Project. My research interests lie in Human-Computer Interaction (HCI) and ubiquitous computing with an emphasis on hardware and sensing technologies. More specifically, I am interested in developing interactive systems for mobile devices, and developing new hardware and sensing technologies to support user interactions in mobile/ubiquitous computing environments. I am also interested in understanding user practices and uncovering problems in difficult interaction contexts through quantitative and qualitative approaches. Besides HCI and ubiquitous computing, I am interested in machine learning, computational linguistics, statistics, psychology, and physiology.
I am looking for passionate research interns who like challenging themselves to hard-but-interesting problems. If you are interested in coming to the most vibrant city in the world, and working on the following topics (but not limited to), please feel free to drop me an email. hacking hardware and developing new sensing technologies, exploring new output modalities (such as haptics, temperature, smell, etc), improving interfaces on mobile/small devices, developing an interactive system using computational linguistics methods (such as text summarization, translation, or information retrieval), and building an intelligent interface using machine learning techniques to handle large-scale data. Contact Office: Microsoft Asia Beijing West Campus, Tower 2, No. 5, Dan Ling St., Haidian District, Beijing, PRC, 100080 e-mail: [my_first_name]@microsoft.com

Context menus, most commonly the right click menu, are a traditional method of interaction when using a keyboard and mouse. Context menus make a subset of commands in the application quickly available to the user. However, on tabletop touchscreen computers, context menus have all but disappeared. In this paper, we investigate how to design context menus for efficient unimanual multi-touch use. We investigate the limitations of the arm, wrist, and fingers and how it relates to human performance of multi-targets selection tasks on multi-touch surface. We show that selecting targets with multiple fingers simultaneously improves the performance of target selection compared to traditional single finger selection, but also increases errors. Informed by these results, we present our own context menu design for horizontal tabletop surfaces.

Mobile and handheld devices have become platforms to support remote collaboration. But, their small form-factor may impact the effectiveness of the visual feedback channel often used to help users maintain an awareness of their partner's activities during synchronous collaborative tasks. We investigated how visual and tactile feedback affects collaboration on mobile devices, with emphasis on spatial coordination in a shared workspace. From two user studies, our results highlight different benefits of each feedback channel in collaborative handheld systems. Visual feedback can provide precise spatial information for collaborators, but degrades collaboration when the feedback is occluded, and sometimes can distract the user's attention. Spatial tactile feedback can reduce the overload of information in visual space and gently guides the user's attention to an area of interest. Our results also show that visual and tactile feedback can complement each other, and systems using both feedback channels can support better spatial coordination than systems using only one form of feedback.

Context menus, most commonly the right click menu, are a traditional method of interaction when using a keyboard and mouse. Context menus make a subset of commands in the application quickly available to the user. However, on tabletop touchscreen computers, context menus have all but disappeared. In this work, we investigate how to design context menus for efficient unimanual multi-touch use. We investigate the limitations of the arm, wrist, and fingers and how it relates to human performance of multi-targets selection tasks on multi-touch surface. We show that selecting targets with multiple fingers simultaneously improves the performance of target selection compared to traditional single finger selection, but also increases errors. Informed by these results, we present our own context menu design for horizontal tabletop surfaces.

Current soft QWERTY keyboards often consume a large portion of the screen space on portable touchscreens. This space consumption can diminish the overall user experience on these devices. In this work, we present the 1Line keyboard, a soft QWERTY keyboard that is 140 pixels tall (in landscape mode) and 40% of the height of the native iPad QWERTY keyboard. Our keyboard condenses the three rows of keys in the normal QWERTY layout into a single line with eight keys. The sizing of the eight keys is based on users’ mental layout of a QWERTY keyboard on an iPad. The system disambiguates the word the user types based on the sequence of keys pressed. The user can use flick gestures to perform backspace and enter, and tap on the bezel below the keyboard to input a space. Through an evaluation, we show that participants are able to quickly learn how to use the 1Line keyboard and type at a rate of over 30 WPM after just five 20-minute typing sessions. Using a keystroke level model, we predict the peak expert text entry rate with the 1Line keyboard to be 66-68 WPM.

Many people read online reviews written by other users to learn more about a product or venue. However, the overwhelming amount of user-generated reviews and variance in length, detail and quality across the reviews make it difficult to glean useful information. In this work, we present the iterative design of our system, called Review Spotlight. It provides a brief overview of reviews using adjective-noun word pairs, and allows the user to quickly explore the reviews in greater detail. Through a laboratory user study which required participants to perform decision making tasks, we showed that participants could form detailed impressions about restaurants and decide between two options significantly faster with Review Spotlight than with traditional review webpages.

Visually demanding interfaces on a mobile phone can diminish the user experience by monopolizing the user's attention when they are focusing on another task and impede accessibility for visually impaired users. Because mobile devices are often located in pockets when users are mobile, explicit foot movements can be defined as eyes-and-hands-free input gestures for interacting with the device. In this work, we study the human capability associated with performing foot-based interactions which involve lifting and rotation of the foot when pivoting on the toe and heel. Building upon these results, we then developed a system to learn and recognize foot gestures using a single commodity mobile phone placed in the user's pocket or in a holster on their hip. Our system uses acceleration data recorded by a built-in accelerometer on the mobile device and a machine learning approach to recognizing gestures. Through a lab study, we demonstrate that our system can classify ten different foot gestures at approximately 86% accuracy.

We describe techniques for direct pen+touch input. We observe people's manual behaviors with physical paper and notebooks. These serve as the foundation for a prototype Microsoft Surface application, centered on note-taking and scrapbooking of materials. Based on our explorations we advocate a division of labor between pen and touch: the pen writes, touch manipulates, and the combination of pen+touch yields new tools. This articulates how our system interprets unimodal pen, unimodal touch, and multimodal pen+touch inputs, respectively. For example, the user can hold a photo and drag off with the pen to create and place a copy; hold a photo and cross it in a freeform path with the pen to slice it in two; or hold selected photos and tap one with the pen to staple them all together. Touch thus unifies object selection with mode switching of the pen, while the muscular tension of holding touch serves as the glue that phrases together all the inputs into a unitary multimodal gesture. This helps the UI designer to avoid encumbrances such as physical buttons, persistent modes, or widgets that detract from the user's focus on the workspace.

One of the challenges with using mobile touch-screen devices is that they do not provide tactile feedback to the user. Thus, the user is required to look at the screen to interact with these devices. In this paper, we present SemFeel, a tactile feedback system which informs the user about the presence of an object where she touches on the screen and can offer additional semantic information about that item. Through multiple vibration motors that we attached to the backside of a mobile touch-screen device, SemFeel can generate different patterns of vibration, such as ones that flow from right to left or from top to bottom, to help the user interact with a mobile device. Through two user studies, we show that users can distinguish ten different patterns, including linear patterns and a circular pattern, at approximately 90% accuracy, and that SemFeel supports accurate eyes-free interactions.

Some of the most interesting differences between Open Source Software (OSS) development and commercial colocated software development lie in the communication and collaboration practices of these two groups of developers. One interesting practice is that of diagramming. Though well studied and important in many aspects of co-located software development (including communication and collaboration among developers), its role in OSS development has not been thoroughly studied. In this project, we investigate how and why OSS contributors use diagrams in their work. We explore differences in the use and practices of diagramming, their possible reasons, and present design considerations for potential systems aimed at better supporting diagram use in OSS development.

Mobile phones are the most prevalent example of pervasive computing technologies in use today, with phone subscriptions reaching 3.3 billion in 2007. According to a 2005 estimate, consumers discard roughly 125 million mobile phones into landfills every year. Although devices continue to proliferate, viable options for ecologically responsible solutions remain elusive, inaccessible, or unknown to users. We examine people's practices with mobile phones, particularly those surrounding end-of-use. We focus on the differences and commonalities between practices in North America, Japan, and Germany, and the impact of varying local constraints on mobile phone sustainability. Building upon previous research examining sustainability and mobile phone ownership decisions, we explore the notion of situated sustainability by looking at how mobile phone sustainability is affected by local and community factors.

Many mobile devices have touch-sensitive screens that people interact with using fingers or thumbs. However, such interaction is difficult because targets become occluded, and because fingers and thumbs have low input resolution. Recent research has addressed occlusion through visual techniques. However, the poor resolution of finger and thumb selection still limits selection speed. In this paper, we address the selection speed problem through a new target selection technique called Escape. In Escape, targets are selected by gestures cued by icon position and appearance. A user study shows that for targets six to twelve pixels wide, Escape performs at a similar error rate and at least 30% faster than Shift, an alternative technique, on a similar task. We evaluate Escape's performance in different circumstances, including different icon sizes, icon overlap, use of color, and gesture direction. We also describe an algorithm that assigns icons to targets, thereby improving Escape’s performance.

Effective text entry on handheld devices remains a significant problem in the field of mobile computing. On a personal digital assistant (PDA), text entry methods traditionally support input through the motion of a stylus held in the user's dominant hand. In this paper, we present the design of a two-handed software keyboard for a PDA which specifically takes advantage of the thumb in the non-dominant hand. We compare our chorded keyboard design to other stylus-based text entry methods in an evaluation that studies user input in both stationary and mobile settings. Our study shows that users type fastest using the miniqwerty keyboard, and most accurately using our two-handed keyboard. We also discovered a difference in input performance with the mini-qwerty keyboard between stationary and mobile settings. As a user walks, text input speed decreases while error rates and mental workload increases; however, these metrics remain relatively stable in our two-handed technique despite user mobility.

ARHunter is a computer-enhanced multi-player whack-a-mole game. It creates an immersive entertainment environment combined with gestural input and location recognition technologies, which aims at increasing the level of players' engagement and excitement.

We developed a positioning technique using ultrasonic signals. Our technique can accurately identify the relative distance and orientation between devices by using an one-time ultrasonic packet. The technique, which is named phase accordance method, uses two or more carriers in ultrasonic communication. A special ultrasonic burst signal, called a sync pattern in the header part of the communication packet gives the base point of the time measurement. The whole time difference calculation is then carried out using this base point. An experiment showed that the technique yielded errors of less than ± 1 mm for 3 m distance measurements and less than 0.5 degree errors for smaller than 30 degree.

Toss-It provides intuitive information transfer techniques for mobile devices, by fully utilizing their mobility. A user of Toss-It can send information from the user's PDA to other electronic devices with a toss or swing action, as the user would toss a ball or deal cards to others. Toss-It uses inertial sensors and optical markers to recognize the user's gestures and location.

We developed a system called Pi_book to support children's exploration in a science museum. Pi_book provides additional contents about exhibitions with PDAs. Additional contents on the PDAs are designed to be interactive in order to increase the children's interests in exhibitions and contents on the PDAs. Our system help children have interests scientific phenomina which is often difficult to understand without any assistance.

Musex supports children's collaborative learning and exploration in a museum with PDAs (Personal Digital Assistants). Musex provides questions about exhibitions that are not interactive, such as explanatory panels and videos. In this manner, our system encourages children to look into these exhibitions. Our user study with Musex revealed that children interacted with exhibitions actively and were engaged in solving questions with Musex.