Tag Archives: research

FABulous CHI 2016

At the CHI 2016 conference this week, there were a slew of presentations on the topic of fabrication. First of course I have to highlight our own Megan Hofmann who presented our paper, Helping Handsa study of participatory design of assistive technology that highlights the role of rapid prototyping techniques and 3D printing. In addition, Andrew Spielberg (MIT & Disney Intern) presented RapID (best), a joint project with Disney Research Pittsburgh which explores the use of RFID tags as a platform for rapidly prototyping interactive physical devices by leveraging probabilistic modeling to support rapid recognition of tag coverage and motion.

I was really excited by the breadth and depth of the interest at CHI in fabrication, which went far beyond these two papers. Although I only attended by robot (perhaps a topic for another blog post), attending got me to comb through the proceedings looking for things to go to — and there were far more than I could possibly find time for! Several papers looked qualitatively at the experiences and abilities of novices, from Hudson’s paper on newcomers to 3D printing to Booth’s paper on problems end users face constructing working circuits (video; couldn’t find a pdf) to Bennett’s study of e-NABLE hand users and identity.

There were a number of innovative technology papers, including Ramaker’s Rube Goldbergesque RetroFab, Groeger’s HotFlex (which supports post-processing at relatively low temperatures), and Peng’s incremental printing while modifying. These papers fill two full sessions (I have only listed about half).   Other interesting and relevant presentations (not from this group) included a slew of fabrication papers, a study of end user programming of physical interfaces, and studies of assistive technology use including the e-NABLE community.

Two final papers I have to call out because they are so beautiful: Kazi’s Chronofab (look at the video) and a study in patience, Efrat’s Hybrid Bricolage for designing hand-smocked artifacts (video, again can’t find the paper online).

AMIA trip report

I have been curious about AMIA for some time, and was even invited to be part of a panel submission to it this year. So when I realized it was only a few hours’ drive away, I took advantage of the closeness to plan a last minute trip. It has been an interesting experience and well worth the attendance. Although a very large conference, the group of people attending seems to be friendly and open, and I was welcomed in particular by two wonderful women I met, Bonnie Kaplan and Patti Brennan. The sessions are an intriguing combination of computer science, medicine, and clinical practice (with the quality of/knowledge about each varying based on the expertise/presence of appropriate collaborators).  I attended sessions on Monday, Tuesday, and Wednesday. The theme that stood out to me more than any other across my experiences here was the great diversity of stakeholders that are (and especially that should be) considered in the design of effective health IT. Some very interesting observations came out of the large scale analysis of clinical data that I saw discussed on Monday. For example, there is a lot of attention being paid to data privacy (although one person commented that this is commonly misunderstood as “Uniqueness is not synonymous with being identified”) and particularly how to scrub data so that it can “get past IRB” for further analysis. One interesting approach taken by N. Shah (Learning Practice-based Evidence from unstructured Clinical Notes; Session S22) is to extract the terms (features) and use those instead of the data itself. Of course a limitation here is that you have to think of features ahead of time.

Another interesting topic that came up repeatedly is the importance of defining the timeline of the data as much as the timeline of the person. Questions that need to be answered include what is time zero in the data being analyzed (and what might be missing as a result); what is the exit cause, or end moment for each person in the database (and who is being left out / what is the bias as a result?); and the observation that in general “sick people make more data.” To this I would add that if you attempt to address these biases by collecting information there is potentially selection bias in the subjects and the impact of the burden of sensing on the data producer. Connected to this is the ongoing questions of the benefits and problems of a single unique identifier as a way of connecting health information.

Last observation from Monday is the question of what public data sets are out there that we should make ourselves aware of. For example, MIT has big data medical initiative and (also see http://groups.csail.mit.edu/medg/) and may have a clinical notes data set associated with it (I am still looking for this).

On Tuesday I started the day with S44: year in review  (D. Masys). I missed the very start of it, but came in when he was talking about studies of IT’s use in improving clinical practice, such as a study showing that reminding clinicians to do their work better improves patient outcomes (“physician alerts” “embedded in EHR systems” etc), or maybe just improves process, with the observation that we should measure both. Interestingly to me, the question of also improving process and outcomes by organizing the work of caregivers (and reminding them of things) was missing from this discussion.

Dr. Masys then moved on to explore unexpected consequences of IT that had been published: adding virtual reality caused “surgeon blindness” to some information; missed lab results in another study and alert fatigue in another (drug-drug interactions suffer from 90% overrides…). Given the difficulty of publishing negative results, it would be interesting to explore this particular set of work for tips. It was also interesting to hear his critique of questionable results, particularly the repeated mentions of hawthorne effects  because so many interventions are compared to care as usual (rather than an equal-intensity control condition). Another way of phrasing this is to ask at what cost does the intervention work (and/or how do we “adjust for the intensity of the intervention” )

Another category Dr. Masys explored of interest to me was health applications of mobile electronics. Briefly, one study looked at chronic widespread pain … reduced ‘catastophizing’; four looked at text messaging vs telephone appointment reminders; effectiveness of a short message reminder in increased followup compliance; text4baby mobile health program; cameroon mobile phone SMS 9CAMPS) trial (PLoS One)

Dr. Masys then moved on to the practice of clinical informatics and bioinformatics (out of “the world of rcts”). This focused on new methods that might be interesting. I particularly want to follow up on one of the few studies that looked at multiple stakeholders which had the goal of reducing unintended negative consequences; the use of registries to do low cost, very large trials;  the use of a private key derived from dna data being encrypted for encrypting that same data; and the creation of a 2D barcode summarizing patient genetic variants that affect the dose or choice of a drug; and a demonstration that diagnostic accuracy was as good on a tiny mobile phone screen as a big screen.

The last category reviewed by Dr. Masys was editors choice publications from JAMIA; J. of Biomed. Informatics; and the Diane Forsyth award. Almost all of these seem worth reviewing in more depth — particularly the JAMIA articles scientific research in the age of omics (explores the need to increase accountability of scientists for the quality of their research) web-scale pharmacovigilance (used public search engine logs to detect novel drug drug interactions); CPOEs decrease medication errors (a meta study that basically concluded without realizing it that CPOEs would work better if we had only applied basic principals from contextual inquiry!) and the JBI articles by Rothman, who developed a continuous measure of patient condition-predicted hospital re-admission and mortality independent of disease (how does this compare with patient reported health status); Weiskopf (who documented the relative incompleteness of EHR data across charts he studied); Friedman’s overview of NLP state of the art and prospects for significant progress (summary of a workshop); Post’s article on tools for analytics of EHR data; and Valizadegan’s article on learning classification models from multiple experts who may disagree (given my interest in multiple viewpoints).

Next, I attended a panel about Diana Forsyth (obit; some pubs; edited works), an ethnographer who had a big impact on the field of medical informatics (and others as well) … she has passed away, and perhaps only a small number of people read work, but her work had an enormous influence on those people who encountered her writing on methods, research topics, and so on. She was compared by one panelist to Arthur Kleinman (who helped to make the distinction between the abstraction of disease and the human experience of illness; treatment and healing). Some of the most interesting parts of the discussion were focused on how the field is changing over time, prompted by a question of Katie Siek’s — for example getting data into the computer, computers into the hospitals, now making them work for people correctly, and what comes after that? Another interesting comment was about the authority of the physician being in part based on their ability to diagnose (which conveys all sorts of societal benefits). This points to the role of the physician (when diagnosis doesn’t exist human creativity is especially needed) versus IT (which can handle more well defined situations). However with respect to healing, maybe the power of physicians is in listening as much as diagnosing (also something computer’s can’t do, right?). Other topics that came up included the importance of the patient voice and patient empowerment/participation.

After lunch with a friend from high school I attended S66 (User centered design for patients and clinicians). In line with the hopes of the Forsyth panel I saw a mixture of techniques here including qualitative analysis. Unfortunately, what I did not see was technology innovation (something that may point to a different in vocabulary regarding what “user centered design” means). However the qualitative methods seemed strong. One interesting talk explored the issues in information transfer from the hospital to home health care nurses. A nice example of some of the breakdowns that occur between stakeholders in the caregiver community. More and more, however, I find myself wondering why so much of the work here only focuses on caregivers with degrees of some sort in medicine (as opposed to the full ecology of caregivers). I was pleased to see low-income settings represented, exploring the potential of mobile technology to help with reminders to attend appointments and other reminders; and a series of 3 studies on health routines culminating in a mobile snack application (published at PervasiveHealth) by Katie Siek & collaborators. One nice aspect of this project was that the same application had differing interfaces for different stakeholders (e.g. teenagers vs parents).

I started to attend the crowdsourcing session after the break, but it did not appear to have much in terms of actual crowdsourcing. An open area for health informatics? Instead I went on to S71, family health history & health literacy. The most interesting paper in the session, to me, looked at health literacy in low SES communities (by many co-authors including Suzanne Bakken). In particular, they have data from 4500 households which they would like to visualize back to the participants to support increased health literacy. Their exploration of visualization options was very detailed and user centered and resulted in the website GetHealthyHeights.org (which doesn’t seem to be alive at the moment). However I have concerns about the very general set of goals with respect to what they hope people will get out of the visualizations. It would be interesting to explore whether there’s a higher level narrative that can be provided to help with this. Similarly, does it make sense to present “typical” cases rather than specific data.

On Wednesday I began in S86: Late breaking abstracts on machine learning in relation to EMRs. This session had some interesting exploration of tools as well as some patient focused work. One study looked at prediction of mobility improvements for older adults receiving home health care, by subgrouping 270k patients and looking at factors associated with the subgroups. Steps included de-identification; standardize data; accounting for confounding factors; divide into sub groups; and then used data mining to look at factors that affected individual scores and group scores using clustering and pattern mining. An interesting take on what is part of the data “pipeline” that goes beyond some of the things I’ve been thinking are needed for lowering barriers to data science. Another looked at decision support for pre-operative medication management (an interesting problem when I consider some of the difficulties faced by the many doctors coordinating my mother-in-law’s care recently).  This work was heuristic in nature (a surprising amount of work here is still focusing on heuristics over other more statistically based approaches). From this work I noticed another trend however, the need to connect many different types of information together (such as published work on drugs, clinical notes, and patient history).

The last session I attended was S92, one of the few sessions focused specifically on patients (and not very well attended…). The first talk was about creating materials for patient consumption, supporting access to EHRs, 2-way secure messaging, and customized healthcare recommendations. They focused especially on summarizing medication information concisely. The second is about a national network for comparative effectiveness. Maybe this is the crowdsourcing of health IT? This was focus group based research (a surprising popular method across AMIA given how little support there is for this method in HCI) exploring user attitudes about data sharing. Interesting that the work presented here ignored a long history of research in trust in computing e.g. from Cliff Nass, the e-commerce literature, and so on. However, the data was nicely nuanced in exploring a variety of ethical issues and acknowledging the relative sophistication of the group members with respect to these issues. The issues raised are complex — who benefits, who owns the data, how would the bureaucracy function, how to manage authorization given that studies aren’t always known yet (and opt-in vs opt-out approaches). I wonder how a market for research would function (think kickstarter but I donate my data instead of money…). The next paper looked at what predicted people thinking EHR are important both for themselves and their providers, and through a disparities lens.

The closing plenary was given by Mary Czerwinski (pubs) from Microsoft Research. I always enjoy her talks and this was no exception. Her focus was on her work with affective systems, relating to stress management. Her presentation included the a system for giving clinicians feedback about their empathy in consults with patients; as well as a system for giving parents reminders when they were too stressed to remember the key interactions that could help their ADHD kids. Interestingly, in the parent case, (1) the training itself is helpful and (2) the timing is really important — you need to predict a stress situation is building to intervene successfully (I would love to use this at home :). She ended by talking about a project submitted to CHI 2014 that used machine learning to make stress management suggestions based on things people already do (e.g. visit your favorite social network; take a walk; etc). One of the most interesting questions was whether emotional state could predict mistake making in coding data (or other tasks).

Would I go back a second time? Maybe … It is a potentially valuable setting for networking with physicians; the technical work is deep enough to be of interest (though the data sets are not as broad as I’d like to see). It’s a field that seems willing to accept HCI and to grow and change over time. And the people are really great. The publishing model is problematic (high acceptance rates; archival) and I think had an impact on the phase of the 3421work that was presented at times. What was missing from this conference? Crowdsourcing, quantified self research, patient websites like PatientsLikeMe, patient produced data (such as support group posts), significant interactive technology innovation outside the hospital silo. In the end, the trip was definitely worthwhile.

Some observations about people who might be interesting to other HCI professionals interested in healthcare. For example, I noticed that MITRE had a big presence here, perhaps because of their recent federally funded research center. In no particular order here are some people I spoke with and/or heard about while at AMIA 2013:


Patti Brennan (some pubs) is the person who introduced me to or told me about many of the people below, and generally welcomed me to AMIA. She studies health care in the home and takes a multi-stakeholder perspective on this. A breath of fresh air in a conference that has been very focused on things that happen inside the physician/hospital silo.

Bonnie Kaplan is at the center for medical informatics in the Yale school of medicine. Her research focuses on “Ethical, legal, social, and organizational issues involving information technologies in health care, including electronic health and medical records, privacy, and changing roles of patients and clinicians.”

Mike Sanders from www.seekersolutions.com, which is providing support for shared information between nurses, caregivers & patients, based in B.C. (Canada).

Amy Franklin from UT Health Sciences Center, has done qualitative work exploring unplanned decision making using ethnographic methods. Her focus seems to be primarily on caregivers, though the concepts might well transfer to patients.

Dave Kaufman is a cognitive scientist at ASU who studies, among other HCI and health including “conceptual understanding of biomedical information and decision making by lay people.”  His studies of mental models and miscommunication in the context of patient handoff seem particularly relevant to the question of how the multi-stakeholder system involved in dealing with illness functions.

Paul Tang (Palo Alto Medical Foundation) is a national leader in the area of electronic health records and patient-facing access to healthcare information.

Danny Sands (bio; some pubs)– doctor; entrepreneur; founded society for participatory medicines; focus on doctor-patient communication and related tools; studies of ways to improve e.g. patient doctor email communication.

Dave deBronkart (e-patient Dave, who’s primary physician was Dr. Sands during his major encounter with the healthcare system), best summarized in his Ted talk “Let Patients Help” (here’s his blog post on AMIA 2013)George Demiris from University of Washington studies “design and evaluation of home based technologies for older adults and patients with chronic conditions and disabilities, smart homes and ambient assisted living applications and the use of telehealth in home care and hospice.”. His projects seem focused on elders both healthy and sick. One innovative project explored the use of skype to bring homebound patients into the discussions by the hospice team.
Mary Goldstein who worked on temporal vision of patient data (KNAVE-II) and generally “studies innovative methods of implementing evidence-based clinical practice guidelines for quality improvement” including decision support.

Mark Musen studies “mechanisms by which computers can assist in the development of large, electronic biomedical knowledge bases. Emphasis is placed on new methods for the automated generation of computer-based tools that end-users can use to enter knowledge of specific biomedical content.” and has created the Protégé knowledge base framework and ontology editing system.

Carol Friedman does “both basic and applied research in the area of natural language processing, specializing in the medical domain” including creating the MedLEE system (“a general natural language extraction and encoding system for the clinical domain”). Her overview of NLP paper was mentioned in the year in review above.

Suzanne Bakken (pubs) has been doing very interesting work in low income communities around Columbia in which she is particularly interested in communicating the data back to the data producers rather than just focusing on its use for data consumers.Henry Feldman (pubs) who was an IT professional prior to becoming a physician has some very interesting thoughts on open charts, leading to the “Open Notes” project

Bradley Malin (pubs) is a former CMU student focused on privacy who has moved into the health domain who is currently faculty at Vanderbilt. His work provides a welcome and necessary theoretical dive into exactly how private various approaches to de-identifying patient data are. For example, his 2010 JAMIA article showed that “More than 96% of 2800 patients’ records are shown to be uniquely identified by their diagnosis codes with respect to a population of 1.2 million patients.”


Jina Huh
 (pubs) studies social media for health. One of her recent publications looked at health video loggers as a source of social support for patients. She shares an interest with me in integrating clinical perspectives into peer-produced data.
Katie Siek (pubs) who recently joined the faculty at Indiana does a combination of HCI and health research mostly focusing on pervasive computing technologies. One presentation by her group at AMIA this year focused on a mobile snacking advice application that presented different views to different stakeholders.
Madhu Reddy (some pubs) trained at UC Irvine under Paul Dourish and Wanda Pratt and brings a qualitative perspective to AMIA (he was on the Diana Forsyth panel for instance). He studies “collaboration and coordination in healthcare settings”
Kathy Kim who spoke in the last session I attended about her investigations of patient views on a large data sharing network to support research, but also does work that is very patient centered (e.g. mobile platforms for youth).
Steve Downs who works in decision support as well as policy around “how families and society value health outcomes in children”
Chris Gibbons (some pubs) who focuses on health disparity (e.g. barriers to inclusion in clinical trials and the potential of eHealth systems).

Data Collection & Analytics Tools?

I have become fascinated recently with the question of the role that data has in supporting analysis, action, and reflection. Actually, it would be more accurate to say that I’ve become aware recently that this is an intrinsic driver in much of the work I do, and thus it has become something I want to reflect on more directly. In this post, I want to explore some of the tools others have already built that might support analytics, machine learning, and so on. If you know of something I’ve missed, feel free to share it in the comments! So, in no particular order:

  • Hazy  provides a small handful of key primitives for data analysis. These include Victor, which “uses RDBMS to solve a large class of statistical data analysis problems (supervised machine learning using incremental gradiant algorithms) and WisCi (/ DeepDive, it’s successor), which is “an encylopedia powered by machines, for the people. ” RapidMiner is a similar tool that has been used by thousands. It is open source and supports data analysis and mining.
  • Protégé is “a suite of tools to construct domain models and knowledge-based applications with ontologies” including visualization and manipulation
  • NELL learns over time from the web. It has been running since 2010 and has “accumulated over 50 million candidate beliefs.”  A similar system is
  • Ohmage and Ushahidi are open source citizen sensing platforms (think citizen based data collection). Both support mobile and web based data entry. This stands in contrast to things like Mechanical Turk which is a for-pay service, and games and other dual-impact systems like PeekaBoom (von Ahn et al.) which can label objects in an image using crowd labor, or systems like Kylin (Hoffmann et al.) which simultaneously accelerates community content creation and information extraction.
  • WEKA and LightSide support GUI based machine learning (WEKA requires some expertise and comes with a whole textbook, while LightSide is built on WEKA but simplifies aspects of it, and specializes in mining textual data). For more text mining support, check out Coh-Metrix, which “calculates the coherence of texts on a wide range of measures. It replaces common readability formulas by applying the latest in computational linguistics and linking this to the latest research in psycholinguistics.” Similarly, LIWC, which supports linguistic analysis (not free) by providing a dictionary and a way to compare to that dictionary to analyze the presence of 70 language dimensions in a new text from negative emotions to casual words.

Deployed tools research and products aside, there is also a bunch of research in this area, ranging from early work such as aCappela (Dey et al.), Screen Crayons (Olsen, et al.). More recently, Gestalt (Patel et al.“allows developers to implement a classification pipeline” and Kietz et al. use an analysis of RapidMiner’s many data analysis traces to automatically predict optimal KDD-Workflows.

Luis von Ahn, Ruoran Liu and Manuel Blum Peekaboom: A Game for Locating Objects in Images In ACM CHI 2006

Hoffmann, R., Amershi, S., Patel, K., Wu, F., Fogarty, J., & Weld, D. S. (2009, April). Amplifying community content creation with mixed initiative information extraction. In Proceedings of the 27th international conference on Human factors in computing systems (pp. 1849-1858). ACM.

Dey, A. K., Hamid, R., Beckmann, C., Li, I., & Hsu, D. (2004, April). a CAPpella: programming by demonstration of context-aware applications. InProceedings of the SIGCHI conference on Human factors in computing systems (pp. 33-40). ACM.

Olsen Jr, Dan R., Trent Taufer, and Jerry Alan Fails. “ScreenCrayons: annotating anything.” Proceedings of the 17th annual ACM symposium on User interface software and technology. ACM, 2004.

Kayur Patel, Naomi Bancroft, Steven M. Drucker, James Fogarty, Andrew J. Ko, James A. Landay: Gestalt: integrated support for implementation and analysis in machine learning. UIST 2010: 37-46

Kietz et al. (2012). Designing KDD-Workflows via HTN-Planning, 1–2. doi:10.3233/978-1-61499-098-7-1011

Search and Rescue and Probability Theory

A man and a dog together belaying down a rock face
Canine Search and Rescue (photo from AMRG website)

I spent a fascinating evening with the Allegheny Mountain Rescue Group today. This is a well run organization that provides free help for search and rescue efforts in the Pittsburgh area and beyond. I was in attendance because my kids and I were looking for a way to give Gryffin (our new puppy) a job in life beyond “pet” and we love to work in the outdoors. Canine search and rescue sounded like a fascinating way to do this and we wanted to learn more. During the meeting, I discovered a team of well-organized, highly trained, passionate and committed individuals that has a healthy influx of new people interested in taking part and a strong core of experienced people who help to run things. The discussions of recent rescues were at times heart rending, and very inspiring.

Later in the evening during a rope training session I started asking questions and soon learned much more about how a search operates. I discovered that about a third of searches end in mystery. Of those for which the outcome is known, there is about an even split between finding people who are injured, fine, or have died. Searches often involve multiple organizations simultaneously, and it is actually preferable to create teams that mix people from different search organizations rather than having a team that always works together. Some searches may involve volunteers as well. A large search may have as many as 500 volunteers, and if the target of the search may still be alive, it may go day and night. Searches can last for days. And this is what led me to one of the most unexpected facts of the evening.

I asked: How do you know when a search is over? And the answer I got was that a combination of statistics and modeling is used to decide this in a fairly sophisticated fashion. A search is broken up into multiple segments, and a probability is associated with each segment (that the person who is lost is in a segment). When a segment is searched, the type of search (human only, canine, helicopter, etc.) and locations searched, along with a field report containing details that may not be available otherwise are used to update the probability that a person is in that segment (but was missed) or absent from that segment. Finally, these probabilities are combined using a spreadsheet(s?) to help support decision making about whether (and how) to proceed. According to the people I was speaking with, a lot of this work is done by hand because it is faster than entering data in and dealing with more sophisticated GIS systems (though typically a computer is available at the search’s base, which may be something like a trailer with a generator). GPS systems may be used as well to help searchers with location information and/or track dogs.

Some of the challenges mentioned are the presence of conflicting information, the variability in how reliable different human searchers are, the fact that terrain may not be flat or easily represented in two dimensions, the speed of computer modeling, the difficulty of producing exact estimates of how different searchers affect the probability of finding someone and the variable skill levels of searchers (and the need to organize large numbers of searchers, at times partly untrained). When I raised the possibility of finding technology donations such as more GPS systems, I was also told that it is critical that any technology, especially technology intended for use in the field, be ultra simple to use (there is no time to mess with it), and consistent (i.e. searchers can all be trained once on the same thing).

Although this blog post summarizes what was really just a brief (maybe hour long) conversation with two people, the conversation had me thinking about research opportunities. The need for good human centered design is clear here, as is the value of being able to provide technology that can support probabilistic analysis and decision making. Although it sounds like they are not in use currently, predictive models could be applicable, and apparently a fair amount of data is gathered about each search (and searches are relatively frequent). Certainly visualization opportunities exist as well. Indeed, a recent VAST publication (Malik et al., 2011) looked specifically at visual analytics and its role in maritime resource allocation (across multiple search and rescue operations).

But the thing that especially caught my attention is the need to handle uncertain information in the face of both ignorance and conflict. I have been reading recently about Dempster-Shafer theory, which is useful when fusing multiple sources of data that may not be easily modeled with standar probabilities. Dempster-Shafer theory assigns a probability mass to each piece of evidence, and is able to explicitly model ignorance. It is best interpreted as producing information about the provability of a hypothesis, which means that at times it may produce a high certainty for something that is unlikely (but more provable than the alternatives). For example, suppose two people disagree about something (which disease someone has, for instance), but share a small point of agreement (perhaps both people have a low-confidence hypothesis that the problem is a brain tumor) that is highly improbable from the perspective of both individuals (one of whom suspects migraines, the other a concussion).  That small overlap will be the most certain outcome of combining their belief models in Dempster-Shafer theory, so a brain tumor, although both doctors agree it is unlikely, would be considered most probable by the theory.

One next obvious step would be to do some qualitative research and get a better picture of what really goes on in a search and rescue operation. Another possible step would be to collect a data set from one or more willing organizations (assuming the data exists) and explore algorithms that could aid decision making or make predictions using the existing data. Or then again, one could start by collecting GPS devices (I am suer some of you out there must have some sitting in a box that you could donate) and explore whether there are enough similar ones (android + google maps?) to meet the constraint of easy use and trainability. I don’t know yet whether I will pick up this challenge, but I certainly hope I have the opportunity to. It is a fascinating, meaningful, and technically interesting opportunity.

Malik, A., Maciejewski, R., Maule, B., & Ebert, D. S. (2011). A visual analytics process for maritime resource allocation and risk assessment. In the 2011 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 221-230.

Why study the future?

I have asked myself that question numerous times over the last several years. Why years? Because the paper that I will be presenting at CHI 2013 (Looking Past Yesterday’s Tomorrow: Using Futures Studies Methods to Extend the Research Horizon) is the 5th iteration of an idea that began at CHI 2009, was submitted in its initial form to CHI 2011 and 2012, then DIS 2012, Ubicomp 2012, and finally CHI 2013 (and, I think, winner for most iterations of one paper I’ve ever submitted). Each submission sparked long and informative reviews and led to major revisions (in one case even a new study), excepting the last one (which was accepted).

I am telling this story for two reasons. First, I want to explore what drove me to lead this effort despite the difficulty of succeeding. Second, I want to explore what I learned from the process that might help others publishing difficult papers. Continue reading Why study the future?

More technology?

I just came across a call to arms by Kristina Höök, “A Cry for More Tech at CHI!” in Interactions this month. I was so glad to see her writing about this and I hope the article bears fruit. She talks about the ways in which technology can inspire design — I would argue also enable design — and why alternate forms of publications should be given archival value as one way of supporting tech research (such as videos, demos, etc). Imagine if videos at CHI were valued as much as videos at SIGGRAPH!

I don’t want to say much more because I hope you go and read what she had to say, but I will note that her sentiment doesn’t just apply to CHI. How about other conferences? And a question I’ve been asking myself recently — what can I do to encourage more tech in my own research group? The answer, I’ve discovered, is to uncover what technology research means to me. That I will say more about.

As I mentioned in a recent post about my sabbatical goals, I have spent some time recently trying to reposition myself. I am and will always be driven by real-world problems, usually ones that come out of personal experience. However, if that is the sole driving force in my work, why am I not a sociologist? Or a politician? Or a anthropologist? Why don’t I run a non-profit? Why am I a computer scientist? The answer that I keep coming around to is that I like to build re-usable solutions to problems, solutions that are (ideally) bigger than the problem I started with. In addition, I believe that technology has value in part because it can solve problems in new ways, sometimes better ways, if we are innovative about how we use it, rigorous, and willing to push the boundary of what technology is capable of. So I am also driven by a wish to build systems, hard systems, systems that do things that have not been done before or create new ways of doing things. In fact, when I look back over the technical work I’ve done, after years of trying (for every job search, tenure review, and so on) I think I finally have put my finger on the unifying theme in my work.

I have always, in some way or another, built what I think of now as data-driven interfaces. I’m certainly not the first person to use this term. Nor is it confined to my area (HCI). But to me it describes one of the most important roles that technology has to play in the world. Many of the most revolutionary impacts of technology have centered around its ability to show information (think of spreadsheets and visualization tools), share information (think of everything from email to the web to facebook) and process information (think of the work in context-aware and ubiquitous computing, machine learning, and so on). And my own inspiration has been similar — my honors thesis as an undergraduate centered around exposing what was going on inside of programs; my PhD work on managing the uncertainty that arises when recognizing sensed input; my PhD students have developed/are developing tools for building ambient and peripheral displays (a form of visualization), rapidly prototyping and field testing ubicomp apps (providing essential data about what information belongs in the application in the first place), measuring and predicting which users have difficulty with an interface (a type of information processing), handling uncertain input within the user interface toolkit, and sharing information about energy use in the home. Of course when you have a hammer, everything looks like a nail (and there’s many aspects of each of these projects that I left out so they would look more nail-shaped) but I do believe there’s a theme here.

Having recognized the theme, a natural question that I, as a toolkit builder, find myself asking is: What is hard about building these sorts of interfaces, especially in situations where we expect people to use the resulting applications. I think there are many unsolved problems, along the entire pipeline from deciding what data to use all the way through to acting on it in some way. Ideally, this should all be put together, in a fashion that scaffolds the process as much as possible and enables communication of constraints and other information from one end of the pipeline to other and back.

The idea of a pipeline for data-driven, interactive systems leads to a whole host of interesting questions I hope to begin answering. What should be communicated within the system? What should be communicated to the user? How does all of this change the way we construct toolkits at the input level? What about the output level? What abilities might we want to give end users with respect to data-driven interfaces? How do we help people build effective classifiers when they have not studied machine learning? How do we help people to select and integrate visualization techniques? What new sensors can we construct and what should we sense? When and how might we involve people (i.e. the crowd) in gathering, labeling, extracting features from, interpreting, even visualizing information? How do we trade this off with machines? And finally, how does the interactive nature of the end systems affect the way we should answer any of these questions?

 

Sabbatical goals, revisited

I am more than halfway through my sabbatical now, and it seems appropriate to revisit the goals I had for my sabbatical, and possibly set some new ones before it’s too late to make changes. I’m going to start with those original goals (in gray), and add comments and thoughts..

Learn about other ways of thinking through sustainability. I want to take the time to deeply explore my own beliefs about sustainability, cross-cultural understandings of sustainability, and how both relate to my chosen field. I am planning on spending at least an hour a week just thinking and writing and reading about ethical/social/planetary issues relating to sustainability. I am also planning on teaching my course on sustainability in both of my sabbatical locations. Total time commitment: 5-6 hours per week.

Outcome: So far this has been a success, although not exactly as I had imagined. I just completed a ubicomp submission with Indian co-authors based on interviews in India, and a survey deployed in India, U.S. high and U.S. low income communities. It was fascinating to explore this data, and definitely caused me to rethink the role of technology in sustainability. Related to this I also submitted a grant proposal with several other U.S. faculty intended to explore automated techniques for affecting energy use across three different continents. Much of this was made possible by collaborations developed with the wonderful folks at IBM Bangalore. I am also teaching my environmental class here in Zürich, and although that is just starting, it is interesting to see the differences here. Finally, I have been blogging about sustainability in a non-academic fashion in an attempt to explore basic beliefs and perspectives on sustainability from a radically new perspective, and recently wrote an article for Interactions based on my blog post questioning the basic assumptions underlying HCI work in sustainability.  Not sure if it adds up to an hour a week, but overall I’m happy with the progress on this topic so far.

Expand my toolbox. I want to learn more about hardware and machine learning (I’ve posted about this before on this blog). My current plan is to take a class on machine learning (I have a handy virtual one with me, or I can sign up wherever I’m at) and teach myself hardware using slides from a CMU class & hands on experimentation. I figure if I spend 2-3 hours per week on each (in parallel if possible, in series otherwise) I should make good progress on this over the year. Total time commitment: 4-6 hours per week.

Again, this has been a success. I completed the Stanford Online Machine Learning class this fall and have been leveraging what I learned in my student advising. Overall, I found the class to be enlightening but the homework a little too easy to solve without complete understanding. Still, it did help deepen my understanding of the algorithms and the methods for improving accuracy and so on.  It was highly focused on statistical techniques, and is nicely complemented by the second course I am taking (much more slowly), Carolyn Rosé’s applied machine learning. Now that one ML class is done, I am also working on my knowledge of hardware using slides of Scott Hudson’s and related materials and have gotten through several arduino projects. So not complete, but I’m pretty happy with this. Extra bonus: I can squeeze some hardware work into family time as the kids love to be included in it. 

Finish hanging projects. I have: Three projects that require analysis only and two-three projects that require writing code. I plan on doing these for the most part in series, unless I am able to recruit local talent to help with the latter two. It’s possible they won’t all get done, but I hope at least some will! Estimated time commitment: 4-6 hours per week. Start new projects that I’ve already thought about. I have two in mind. Estimated time commitment: 4-6 hours per week if done in series.

I wish I’d written down which specific projects I intended to work on! For the most part this has not been a great success. I have completed one (writing) project, and recruited students to work on others. However, recruiting students in Switzerland has not worked out well, and I’ve had varying success reaching any complete goal with students I recruited in India. There is still time to accomplish this goal, but I think that I am unlikely to get anywhere near 6. For my own sake when looking back on this six months from now, I will name the projects this time around: Futures — done; Search tools — making progress; Cosmo/viewpoint extraction — making progress; Mechanical Turk web accessibility — stagnating; Diabetes + Lyme analysis — stagnating [but I could tackle this without additional students]; Macro energy audits — stagnating; Using routines to reduce energy use — done; Lo-fi presence — making progress.

Write a large NSF proposal [already started]. Estimated time commitment: 1 hour per week through November.

Sadly petered out at the last minute, though I cannot take the blame for this. On the upside I led a group of five other PIs in writing a medium proposal (mentioned above). It was a fascinating experience in herding faculty which I’m not sure I ever want to repeat! :). 

Continue supporting students. Estimated time commitment: 3-4 hours per week of meetings, 1 hour per week of prep & planning. Meet new people, start new projects, develop new ideas. Estimated time: 4-6 hours per week.

Both successes (though you might ask my current students if they agree :P). Amazing how Skype can shorten the distance between places. I’m now holding meetings across three continents each week, a sign of the new collaborations I’ve begun. Also along the way I’ve given 7 talks and counting. But what I’m most pleased about in this arena is the opportunity that it’s given me to rethink my own research agenda and try to explore new ways of positioning myself. I have begun to realize that while I am driven by applications that matter this has in some ways obscured other things that I care about. It has also affected my ability to recruit a certain type of student — folks for whom programming and building things and solving hard technical problems is as important as the applications this enables. So I have, through public speaking engagements, been exploring a new way of presenting my work, one that emphasizes both the enabling middleware that can make possible the creation of applications that address real world problems using technology. More on this in a future post, as this is running long, but I consider the opportunity to rethink research approaches, goals, and so on to be a major benefit of sabbatical. 

So what next? Certainly, at a minimum more of the same. What I’ve been trying to do is working, and I plan to continue for the next few months. I still need to forge stronger collaborations with some swiss colleagues, and am actively working on that. I also need to step back and ask which hanging projects are worth pursuing and what the best approach to doing that is. And of course I need to keep in mind that this is my chance to rest and rejuvenate.  I will have been deeply involved in writing about 120 pages of text by the end of April, taught most of 4 classes since the start of last summer, and advised or mentored about 13 people (in one-on-ones) across two continents throughout most of the sabbatical, and learned 1.5 new foreign languages. In between, I’ve also been taking time to travel, relax, spend time with my kids, and continue fighting for my health. I plan to continue that, as I wish to arrive home not only inspired and educated but also healthy and well rested for the start of the fall semester.  

A different academic model

Zurich in the snow, from our apartment
Zurich in the snow, from our apartment

The next phase of our sabbatical is in Zürich, Switzerland, where we’ve been since the beginning of January. There hasn’t been much to post here because, I suppose, things feel so familiar. We have a “group” to be part of, thanks to our wonderful host, Friedemann Mattern, which makes a big difference in how integrated we are into the university community. The university setting itself is much more familiar somehow than in India, perhaps for the same reason: We had to work to ensure that our office was near that of other faculty, and actively pursue integration with the department in Hyderabad. Here, we still have to actively pursue potential collaborations, but this is facilitated by the support that Friedemann and his group have given us.

ETH is also familiar in the sense that it functions like most other universities I’ve been part of over the years, as a homing ground for students, an organizer of talks on a wide breadth of topics, a place to discuss and teach and learn. One thing that differs from american universities is the structure of the department. The model here is one person per area. For example, a friend at the University of Zürich is the only person in Human Computer Interaction in her department, and is expected to carry the entire field. Critical mass is built across all of computer science, not within sub-areas. Instead, one recruits a productive and diverse set of post docs, doctoral students, masters students, and so on who work together to make the area a success. This is the polar opposite of a place like Carnegie Mellon, where entire departments are formed around sub-fields.

One of the more interesting things about being on sabbatical is the opportunity to rethink and think through who I am as a researcher. I am frequently given the opportunity to speak about my work to a variety of audiences, and I have written a number of different talks over the year attempting to summarize my work in assistive technology, my work in sustainability, overarching themes for the technical aspects of my work, and deeper questions about the value of the projects that I have chosen to do. Along the way, I have studied machine learning (I will have to write about this, as I took the Stanford ml course last fall) and am now studying hardware in more depth, finally finished a paper on the value of futurism (or rather Futures Studies) in guiding research (an enormous stretch for me, as it is primarily what I would consider a design/thought paper) and an article for interactions questioning the focus of sustainable human computer interaction research, based on a recent blog post on the topic.

To me, the ability to see and think about new models for academia as a whole, my own research, and everything in between, is one of the most valuable things about this year away. It’s a chance to rethink, question, and consider what works, what should be done, and what will make a difference.

Scanning books to explore the future role of technology with respect to climate change

I have been reading up on the discipline of futurism, which academically speaking provides methodological hints for exploring what may happen in the future. One of those approaches, monitoring, and in particular environmental scanning (looking for signs or indicators in large volumes of relevant information) can be of value [1].

Inspired by that and the work of Dourish & Bell in “Reading science fiction alongside ubiquitous computing,” I have begun to work my way through a collection of science fiction, science, and non-fiction books that look forward into the future with respect to climate change. In choosing books, I explored a combination of indie fiction, activist monologues, mainstream science fiction, and scientific writing. In reading these books, I am particularly focusing on how they portray science (or what they say about it) and its interaction with other trends.

I have not had time to read many of the books I’ve found yet, but I have started on a few: Forty Signs of Rain (Robinson) depicts scientists at the NSF in the near future as they try to rethink the role of the organization as the climate reaches a tipping point; The Ultimate Choice (Hinsley) depicts a society in which population growth has eaten up the land needed to grow food and the government is forced to watch its people slowly starve or take more drastic measures; Smart Mobs: The Next Social Revolution (Rheingold) discusses the interaction between technology and the social nature of human society (but has only one paragraph that mentions climate change); The Windup Girl (Bacigalupi) portrays a world already changed for the worse by genetic engineering but relatively untouched by climate change; Flood (Baxter) posits a world that is slowly drowning in oceans rising at an exponentially increasing weight (non-anthropogenic climate change).

Some of these are books I just happened to read recently for pleasure or research while others were selected specifically for this project. All of them take very different perspectives on what may happen with the climate, and equally different perspectives on technology. For example, Robinson’s book is set in a future whose technologies are exactly those of today, while Hinsley posits an Orwellian society in which most citizens are lucky to have food and a map is a technological luxury. Television, of course still exists, but only the state has modern technologies (and they, again, match todays). In Rheingold’s book, the purpose is to explore cutting edge technology; the role of applications, and their interaction with culture and society society (specifically how to avoid threats to liberty, dignity, and quality of life while enabling tho promise of these technologies). Bacigalupi creates a rich portrayal of a world whose technological innovations are new engineered species (semi-human and others) that are at times indistinguishable from (or competitive with) existing species. In Baxter’s world, technology is more prominent than in the other books, despite the fact that technological innovation has (almost) halted (after the creation of the ultimate music player, one which connects almost magically with the brain, without cords) due to the focus on surviving the disastrous consequences of the unstoppable flood. The most prominent technology in the novel is a 3d projection system that can render the earth and illustrate the coming catastrophe. Perhaps equally important is the solar powered mobile phones that allow climate scientists to “convene” virtually once each year as they pay witness to the disastrous flood overtaking the earth.

At this point, I’m still trying to process all of this. I think it is interesting that technology and climate are so divorced in most of these novels. The exception is technology’s role in science in the Robinson and Baxter novels (supporting communication between scientists, visualizations for non-scientists, and so on). None of the books discuss technologies intended to help save energy or otherwise influence behavior; homes are not smart in these books and phones are just phones for the most part. I wonder if the non-fiction books about dealing with climate change are any more likely to mention technology. I suspect not: There is an aspect of luddism to some of the climate non-fiction that would

[1] Bell, W. (2003). Foundations of Futures Studies: Human science for a new era: History purposes and knowledge (Volume 1). Transaction Publishers.

[2] Dourish, P. & Bell, G. (2008). ‘Resistance is Futile’: Reading Science Fiction Alongside Ubiquitous Computing. Personal and Ubiquitous Computing. In Press.

Centre for the Study of Developing Societies

I am in Delhi for the week (mostly with family obligations), but I was able to sneak away for a few hours this afternoon (less time than I hoped) to visit the Center for the Study of Developing Societies, also home to innovative programs likeSarai and IM4Change.

I have been trying to get in touch with them for a few weeks, ever since a collaborator recommended I visit, and it was well worth the trip. I knew about their Hindawi project (which was one of the first attempts to localize the computer experience (specifically, RedHat, because open source meant possible to modify), and that they have a library of social science research.

When I got off the Delhi Metro and walked to their building, I found myself in a quiet area of Delhi. The complex is unassuming and not very guarded (how different from the experience of visiting more high-tech offices). I simply walked in and was taken quickly to the office of the one person I knew a name for, Mr. Ravikant (from the Sarai program). We had an interesting conversation about the Hindawi project and his studies of movies in Indian society, and then I headed off to the library.

The library is about 6 rooms packed with books, and when I arrived also hosted an art exhibit that took over much of the main room (and cut it in half, literally, via a diagonally placed window pane). Despite the whimsy of the room-sized exhibit, the mood in the room was rather serious. The library was run by an older man working on his PhD who pointed me at the room where I might find books on health and sustainability. I had less than an hour to explore it, so I moved quickly to pick out the most relevant books. I came downstairs, arms loaded, and was greeted by a frown and the information that I should be only picking up 4-5 books at a time. Undaunted, I started working my way through my finds only to learn 30 minutes later that the library was closing 15 minutes earlier than advertised.

In the end, I had time only to write down the references for the most interesting books, skim 2 or 3 of them, copy a few quotes, snap photos of a few pages, and so on. The collection was truly unique, with many books published in India or even self-published. I fear I will not be able to find many of these anywhere else. I end this post with a smattering of what I found.

After leaving the library, I ended up walking home beside a member of CSDS who works on media and journalism. His work attempts to translate information needed to improve the accuracy of indian journalism, and we had a long and interesting conversation about localization and stereotypes during my metro ride home (we overlapped for quite a ways). We had a very interesting conversation about his work on IM4Change and indian society overall. We talked about the difficulties I am having in developing an intuition for “india” and the fact that one reason for this is the diversity of who india is and where it is going — this requires a form of localized thinking that echoes the need to think at the individual level in assistive technology. It is not easy to hold on to both the specifics of one environment/context and also the generalities of a whole (and very diverse) set of societies/country. One interesting insight I gained from the conversation is to think about in what ways (and how universally) scarcity defines Indians. Overall, the visit was a reminder that I need to further immerse myself in Indian culture, especially rural culture, if I am to have anything useful to say about those settings. As we’ve learned in HCI, each user is unique. Similarly, in India, each locality is unique (there are 17 languages and over 800 dialects here, just to give a sense of the diversity of different regions across India). While good HCI eventually comes to conclusions that can generalize across users, it is often very effective to first get to know specific users very well. This becomes even more important when the user is foreign to the designer. I don’t know if I’ll get there in my short time here, but at least I have a sense of the road I need to travel now.

Relating to Sustainability:

ML Dantwala (1996). Dillema’s of Growth: The Indian Experience. Sage publications: Looked at agricultural policy; rural development. one chapter concerned with the adoption of high yield seeds.

Conservation for Productive Agriculture. published by the “Publications and Information Division; Indian Council of Agricultural Research; New Delhi. Ed: V.L. Chopra; T.N. Khoshoo: Articles contributed by scientists worldwide. Chapter on water conservation (#7, p. 98)  discusses india’s vast water resources

Rabindranath Mukhopadhyay (1999). Growth of Indian Agriculture: A study of spatial variation. Classical Publishing: Abridged doctoral dissertation. Statistics oriented

Mahmood Mamdani (1972) The Myth of Population Control: Family, Caste and Class in an Indian Village. Monthly Review Press, New York.

B. K. Pradhan, M. R. Saluja, S. K. Singh. Social Accounting Matrix (SAM) for India: Concepts, Construction and Applications. Sage Publications, New Delhi. Detailed modeling (input out put, and SAM ) for workers in india, exploring gender differences, etc.

Sumi Krishna (1996). Environmental Politics: People’s Lives and Development Choices. Sage Publications: Very nicely focused on India and environmentalism. Suggests at the end “Perhaps it is environmentalism that we need to redirect” Raises 4 areas of concern. “First, the environmentalist approach to a range of interlinked inequities, such as gender, caste, and class, seems to be limited because many environmentalists, despite their varied approaches, actually share ‘ideological roots’ with the very paradigms that are sought  to be changed” (e.g. “ecofeminism restricts women just as patriarchal systems do” … “Second, environmentalists have been largely unconcerned with the mechanics of establishing the decentralized institutions (like panchaayats) necessary to achieve sustainability. Centralised environmental management prgorammes … can have little chance of success, if they are conceived by administrators and technocrats, who may themselves be insensitive to the people in the environment … Third, the recognition that commercialization and market forces have a powerful corroding effect on the natural resource base does not mean that we should sentimentalize the traditional interaction of communities with their local environment. … the question is of choice and control – who determines the course of change? and of anticipating the consequences of change. And fourth, environmentalists have had little idea of how to create the necessary ‘value-orientation’. Values operate through interests. The strength of the environmental movement … mobilizing people who do not face an imminent threat is a much more difficult task … how can we build a base of shared values?”

Forests, Environment and tribal economy. Deforestation, impoverishment, and marginalisation in orissa. W. Farnandes, G. Menon, P. Viegas. Indian social institute, tribes of india series.  1988.  Looks at tribes and forests. Tribal traditions, causes of deforestation, consequences, official solutions (10) devolpemnt progammes and people’s solutions (11). Complements agarwal?

D. R. Sha (1985). An economic analysis of co-operative dairy farming in Gujarat.  Somaiya Publicatinos LTD:  Analysis of a specific market economy in gujarat…

Healthcare:

Human Development Report  Uttar Pradesh — Lots of status on health spending & so on. Get the same for AP?

Tribal Health: Socio-Cultural Dimensions (1986). Edited Buddhadeb Chaudhuri. Inter-India Publications, New Delhi: Goes into great depth about medical anthropology in india. has a whole section on interaction of traditional and modern medical practices. Ch. 26 (p.  311- 321) deals specifically with the conflict between traditional medicine and a western program that was implemented, and how these interacted. “This study depicts the failure of state government health programs in motivating the Santals, whereas the health programs initiated through Sriniketan influenced the Santals in a positive way” (no surprise here!). In general the chapters in this book are short.

Sheila Zurbrigg (1984) Rakku’s Story: Structures of ill-health and the source of change. Printed by George Joseph, on behalf of the author! She is a canadian doctor. Sems to be an ethnography, very interesting if I can get it. Available from Centre fro social action, gundappa BLock, 64 Peme Gowda Rd. Bangalore, 560 006 or Sheila Zurbrigg, 1120 the parkway, london, ontario, n6a 2×3 canada.  Talks about the medical model of health and the fact that social factors (food distribution & access for example) have much more impact on overall health than individual disease, and in fact were responsible for improvements in world health before the germ model was even developed.

Om Parkash Sharma (2000). Rural Health and Medical Care in India (A sociological Study).  Manak Publications Study of the “modern healthcare system” in india, focuses on Uttar Pradesh.

J. P. Naik (1977). An alternative system of health care service in india: Some proposals. Allied Publishers Private Limited.

Meera Chatterjee, 1988. Implementing health policy. Manohar,  Centre for Policy Research, New Delhi. Policy focused. Looks at community participation in health care (chapter 5) and the role of the private voluntary health sector (6) spreading primary health technology (7). Life without food? (8).

Not specific to India:

Andrew Jamison. The making of Green Knowledge: Environmental Politics and Cultural Transformation. Cambridge University Press 2001. Not specific to India, but very interesting. Looks at the dilemmas of activism in  chapter 6 p. 147-176

Eco-Phenomonenology: Back to the earth itself. Edited by Charles S. Brown and Ted Toadvine. State University of NY Press. Also not specific to india. Has a chapter on Heiddigger 😛

Ecology, Politics & Violent Conflict. Mohamed Suliman (Ed).  ZED Books, 1998.  International perspective on conflict driven by environmental issues.

Isomäki (r.) and Gandhi (m.) (2004). The book of trees. Other India Press. Global look at trees, written and published in india. seems more manual than introspection.