Penn HCI researchers work to enhance and redefine our relationship with computers and devices.
Modern life involves nearly constant interaction with devices and technology. For users, not only is an understanding of how to perform these interactions with confidence and security important, but an awareness of how those interactions affect us and our communities is critical.
This is true across the full spectrum of our experiences with technology. The recent explosion of large language models like ChatGPT is one example. With such technology increasingly used by everyone from undergraduate students for homework help, to professional programmers writing code for critical systems, to hiring managers reviewing resumes, understanding the situated, real-world use of these tools and their effects is paramount. Answering these questions is the domain of a subfield of computer science called Human-Computer Interaction (HCI).
In the last year, Penn Engineering was fortunate to welcome not one, but two computer scientists with expertise in HCI, launching the School’s Human-Computer Interaction Group. Since, Andrew Head, Assistant Professor in Computer and Information Science (CIS), and Danaë Metaxa, Raj and Neera Singh Term Assistant Professor in CIS, have been seizing the opportunity to advance the HCI Group’s overarching goal of understanding, designing and engineering technologies with the human-centered goal of making a positive impact on individuals and communities.
In broad terms, HCI is a field of study that focuses on improving interactions between people and computers through the design of technology that satisfies users’ wants and needs. “HCI is a discipline founded at the intersection of several fields, including computer science, psychology and media studies, among others,” says Metaxa. “It’s incredibly broad as a subfield, encompassing everything from wearable haptic interfaces to analyses of political persuasion in social media.”
Located within the CIS department, the HCI Group supports research projects that cover a variety of topics spanning the development of new patterns of interaction with computers, refining the interfaces of existing applications, and understanding the effect that computers and their applications have on people.
“In my research, I explore the future of reading,” says Head. “As our computers get better at understanding the texts humans write, we can open the doors to the complex, everyday texts people need to understand. Could our reading interfaces help us understand our medical records? How about tricky math equations, or tangled source code? I explore how modern techniques from AI and program analysis can enhance reading experiences in these settings.”
“In my research,” says Metaxa, “I develop and deploy methods for studying bias and representation in algorithmic content, focusing on high-stakes social settings like politics and employment, and especially on the experiences of marginalized people. This includes the identification of biases in existing systems, as well as the design of interventions and building of new systems that try to remedy those issues.”
In addition, the HCI Group offers a range of courses at the graduate and undergraduate levels, including “Introduction to HCI” and more specialized courses. For example, a graduate seminar titled “Algorithmic Justice” examines a growing body of work at the intersection of technology and social justice. A range of areas are included under this umbrella, including tech ethics, design justice and algorithmic fairness, as well as work on equity, bias, diversity and representation in computer science and other related disciplines.
Spanning the Gamut
In the short time since its inception, the HCI Group has launched a rich portfolio of projects. Within Metaxa’s group, researchers are developing new insights about the nature of representation in online settings, both emerging and established. One illustration of this research is a project titled “Sociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising.” In that work, the team built a system to measure the targeted web ads received by a diverse group of people, and also experimented with showing them different ads, getting at the question of whether surveillance-heavy ad targeting systems actually benefit users. Their findings indicate that much of the efficacy of targeted ads is driven by prior exposure, suggesting that more privacy-preserving policies should be pursued for online ads.
CIS doctoral student Princess Sampson examines the impact of people’s identities on their interactions with algorithmic systems. “Engineering with inclusion and agency in mind enables better algorithmic outcomes for everyone,” says Sampson, who led an empirical study titled “Representation, Self-Determination, and Refusal: Queer People’s Experiences with Targeted Advertising,” which found that LGBTQ+ perspectives highlight the risks of an online advertising industry that doesn’t allow end-users to curate their ads or understand how they’re being targeted.
Inspired by models of community-driven filtering for email and social media, Sampson is now building Admix, a system for people to collaboratively archive and filter their online ads. “We aim to empower people to collectively combat the wide spectrum of issues presented by online ads, on their own terms,” says Sampson.
In Head’s group, researchers are developing tools to help readers understand complex texts. CIS doctoral student Litao Yan explores how tools can help programmers understand code generated by programming assistants like GitHub Copilot. “As AI programming assistants make their way into the editors of millions of programmers, these programmers find themselves with an entirely new obligation,” says Yan. “They must understand the code that their AIs have generated.” Yan developed an extension to a code editor that automatically explains the code that programming assistants generate with AI to keep programmers engaged in understanding the code their programming assistant is generating.
Meanwhile, CIS doctoral student Hita Kambhamettu is exploring how AI-assisted reading tools can help patients understand their health records. “Electronic health records have become a crucial interface in health care for documenting a patient’s medical history and establishing common ground between providers and patients,” she says. Kambhamettu conducted an observational study of patients as they read their records to understand the challenges patients face while reading them. She is now building medical reading interfaces that incorporate AI to explain the jargon and data that appear in health records.
Other students in the group are developing tools to help people understand math equations, computerized proofs and complex instructions. Across the gamut of projects, the theme is to
develop new paradigms for transforming complex texts so that they might be better understood.
Pulled to Penn
Supporting a strong team of talented students was one major reason Metaxa was excited at the opportunity to start a new group at Penn. “When I was an undergraduate at Brown and the Computer Science department made its first HCI hire, I got to see professor Jeff Huang, who advised my senior thesis, build his group from the ground up, and I relish the opportunity to do the same here at Penn,” says Metaxa. “What’s more, I’ve known Andrew since the beginning of our doctoral programs — we were both in California, he at Berkeley and I at Stanford — and the chance to work so closely with him in building this group sounded like, and indeed has proven to be, a wonderful experience.”
According to Metaxa, the strength and interdisciplinary research interests of the faculty in the School are invaluable. “Situated as one of 12 very strong schools within the University, Penn Engineering is a great home for this work since it allows us to draw on strengths from across campus, including in our own department,” says Metaxa. “For instance, I have a secondary appointment in Annenberg. For my Ph.D., I was co-advised by a computer scientist and a communications scholar. Penn allows me to continue having connections in both worlds. And within our department, we have colleagues with co-appointments in a range of areas, including the School of Medicine, Wharton and others.”
“Penn is an amazing place for HCI,” says Head. “Danaë and I have had great agency to build the HCI Group from the ground up, and we now have a home for our research teams where students develop strong roots in our discipline. Many of our students are building bridges into other disciplines like programming languages, health informatics, NLP and communications. This has made for a rich research environment where we work on problems that matter, with the methods it takes to make a difference.”
Head and Metaxa are excited to continue growing HCI. “We opened the doors to HCI in our department, and students have come in. We now regularly welcome students from wide disciplinary backgrounds across the University into our courses,” says Head. “There is nothing more exciting than making spaces for students to critically examine how technology can be developed to better serve people.”
Story by Janelle Weaver / Photo by Jeff Wojtaszek