Why did it make me uncomfortable to tell this story? First, I personally know many members of the SIGCHI EC, and I know how much they care about accessibility, and how much hard work, and change, has already happened thanks to AccessSIGCHI‘s efforts. I worry that my blog post will make them think that these efforts have gone unseen.
Second, I was very much hurt and angered by my experience, and it also made me feel unsafe. Perhaps this worry is misplaced, but by bringing it all up again, and making it more public, it seems possible that folks will think I behaved unethically or even go further in prosecuting this accusation.
However, I believe that taking risks is how we make progress, and I deeply hope that my love for SIGCHI, and for the individual volunteers within and outside of the EC, will help carry the day and allow this challenging moment to create a positive coming together rather than a rift between SIGCHI and its constituent communities.
The Accessibility work undertaken by the EC is very very important. However it must be complemented by equally important work that addresses structural inequity. As I said in my blog post
When work is done by marginalized groups, such as the R.A.C.E. team and AccessSIGCHI, it is especially important to nurture and cultivate their perspectives. The cost of the commitment of individual time to these efforts should not be underestimated, especially given the likelihood that many such individuals are continually being asked to put extra time into representing their community as well as advocating for themselves. Even small blows to these efforts have the potential to eliminate a gift that could otherwise help our community to better itself.
It has been two years since the SIGCHI Accessibility Community first published a report on the state of accessibility within SIGCHI, summarized in [Mankoff, 2016]. This report was the first of its kind for SIGCHI, and reflected a growing awareness in the SIGCHI community of a need to directly engage with and improve the accessibility of both conferences and online resources sponsored by SIGCHI. The report argued that “SIGCHI can attract new members, and make current members feel welcome by making its events and resources more inclusive. This in turn will enrich SIGCHI, and help it to live up to the ideal of inclusiveness central to the concept of user-centered design”and that ACM’s own guidelines clearly argue for a standard that SIGCHI was not meeting.
The report laid out a series of five recommendations for improving the accessibility of SIGCHI, including the accessibility of conferences (R1) and content (R2). Additional recommendations include a better process for handling accessibility requests (R3), increasing representation of people with disabilities within SIGCHI (R4) and assessing success at least once every two years (R5). This update is our attempt to fulfill R5.
The rest of this report starts an executive summary of the biggest accomplishments of the last two years (along with the data sources used to draw those conclusions). The remainder is organized by recommendation, highlighting the goals set out and what has been accomplished.
The SIGCHI Accessibility community has made significant progress over the last two years, particularly on short term goals set out for R1 (conference accessibility), R4 (increasing representation) and R5 (assessing success). Less progress has been made on R2 (content accessibility) and R3 (handling accessibility requests). Most significant among our progress are the release of new accessibility guidelines for conferences collaboratively with SIGACCESS (http://www.sigaccess.org/welcome-to-sigaccess/resources/accessible-conference-guide/), publication of a CACM article on accessibility efforts within SIGCHI [Lazar, 2017], addition of an accessibility representative to the CHI Steering committee, and establishment of a facebook group, sigchi-access
Despite our progress, there are major steps that still need to be taken in every category. In addition, the last two years’ effort has made it clear that the five goals represent a more ambitious program than a small volunteer community can easily achieve all on its own. We are very grateful for the help of SIGACCESS and members of the SIGCHI leadership, which have made a huge difference in what the community was able to accomplish in the last two years. However, as long as we remain an all volunteer effort, the SIGCHI Accessibility community will have to decide what to prioritize over the next two years: Continued effort on R1, or other goals. For these reasons, we strongly recommend that both the SIGCHI leadership and the community consider what funds might be available to help move this effort forward, what tasks can be transferred to contract employees of SIGCHI such as Sheridan, and fundraising efforts to support these changes.
Analysis of SIGCHI Accessibility Metrics
The SIGCHI Accessibility Communities efforts focus on the digital and physical accessibility of SIGCHI resources. As such, important metrics relate to the experiences of people with disabilities who consume those resources as well as the overall percentage of the resources that are accessible. As will become evident in this report, our current focus is on online videos, pdfs, and conference accessibility. Thus, relevant metrics might include, for example, the percentage of new videos that are accessible, the percentage of PDFs that are accessible, the percentage of conferences that have accessibility chairs, and surveys of conference attendees.
For this report, the Accessibility Community combined several different sources of data. As with the previous report, much of our data is focused on conference and meeting participation, and we use the same categories: We refer the reader to the original report for details on these categories. To briefly summarize, we include direct observation, experiences reported by SIGCHI accessibility community members and survey data from conferences (CHI and CSCW 2014-16). Appendix A has the survey questions that were asked. We did not conduct a survey of our own of community members this year.
Results
Overall, there is a trend toward increased accessibility at CHI, based on the conference surveys. However, these numbers are difficult to interpret since the direct question about access needs was not asked in 2016. In addition, it is possible multiple people might report on the same person not being able to attend, some of these answers are about financial barriers rather than disability, the sample is self selected, most people who cannot attend may not be remembered by attendees filling out the survey, and people with disabilities who attend may not want to disclose.
Accessibility questions and answer counts from CHI 2014, 15 and 16 conference surveys.
For all of these reasons, it is in many ways more instructive to look at the categories of issues reported. However, in our analysis, the majority of specific data reported was CHI 2016. Thus, we cannot easily describe trends over time. Here are the major categories mentioned in the open ended responses to the CHI 2016 survey: Difficulty of networking; lack of ramps for presenters; need for speech to text or sign language interpretation; need for accessible digital materials (program, maps, pdfs); distance between rooms; lack of reserved seats in the front of rooms (e.g., for the hearing impaired); cost of attendance for people with disabilities; prioritization of robots over people with disabilities; accessibility of robots; food issues; need for a quiet room or space; bathroom usability.
In addition, there were several issues that have to do with educating the community. These included the need to coach presenters to remember some people cannot see their slides, the importance of an SV orientation training segment for special needs work, the importance of publishing accessibility issues up front (before conference paper submission).
In terms of hard metrics, conference accessibility is increasing. In the chart below, a score of 1 means that either there was a page on the conference website that mentioned accessibility or an accessibility chair of some sort on the organizing committee (2 means both). The Total (bottom bar) is the total number of conferences that year that had at least a score of 1.
Chart showing conferences that mentioned accessibility on their conference website in 2015/16 and 2017/18. The trend is increasing, but the numbers are small.
Video accessibility is also increasing, captions are now included in all videos uploaded through the official SIGCHI Video service (see R2). PDF accessibility is low, and not improving, hovering around 15% [Brady, 2015; Bigham 2016].
R1: Achieve Conference Accessibility
Ensure that 100% of conferences are accessible, have an accessibility policy and have a clear chain of command for addressing accessibility issues.1
The first recommendation from the 2015 report dealt with conference accessibility. The community set out a series of short term goals:
[Not Met] Have an accessibility chair for every conference by the end of 2017. This goal has not been achieved. However, the SIGCHI accessibility community now has representation on the CHI steering committee, and was asked to present to the SIGCHI Executive Committee. Both groups voted to ask all conferences to have a web page documenting their efforts to be accessible (or acknowledging a lack of accessibility), specifically on a /access subpage of their conference URL.
[Met] Educate the community. While there is always more work to be done, the SIGCHI accessibility community has participated in several education efforts, including the publication of an interactions blog post and article [Mankoff, 2016]; a CACM article on making the field of computing more inclusive [Lazar, 2017]; a 2016 SIG on the state of Accessibility within SIGCHI held at CHI [Rode, 2016]; the creation of a facebook group called SIGCHI-access; and the aforementioned presentations to the SIGCHI leadership.
[Met] Create updated conference accessibility guidelines in collaboration with SIGACCESS. This goal was completed, and the guidelines are available at http://www.sigaccess.org/welcome-to-sigaccess/resources/accessible-conference-guide/ In addition to providing general guidance for making a conference accessible, the guidelines provide instructions specifically for how to generate the content that can be placed at the /access URL
To summarize, we have made some progress but not achieved our most important goal of widespread adoption of accessibility practices across conferences. In addition, we have not yet started the longer-term goal of best practices and financial viability. That said, we hope that the /access URL will move us toward the ultimate goal of having all conferences start planning on accessibility from the very beginning.
R2: Achieve Content Accessibility
Ensure that 100% of new content such as videos and papers meets established standards for accessibility and develop a process for achieving this.1
The SIGCHI Accessibility community has not focused as much attention on this goal as on R1 in the last two years. Our short term goals for content accessibility include assessing current status, creating guidelines to use as a standard, and developing a process for addressing accessibility. These goals need to be addressed for multiple types of content — papers, videos, and websites.
The SIGCHI Accessibility community, and others within CHI, have mainly focused on papers and videos in the last few years. No work has been done on websites. With regard to paper accessibility:
[Partly Met] Assess current status. We do not have a comprehensive assessment of current status. However, some statistics on paper accessibility are available in the CACM article [Lazar, 2017]. Specifically, pdf accessibility of CHI papers from 2013 through 2016 is mentioned in that article, and the numbers hover around 15 percent in most years.
[Not Met] Create guidelines to use as a standard. This is an open problem, particularly because in the case of paper accessibility, the most recent ACM revision of the SIGCHI template is less accessible than the 2016 version. SIGACCESS is experimenting with an HTML5 format that SIGCHI may want to adopt if it is successful.
[Not Met] Develop a process for addressing accessibility. Multiple methods for paper accessibility have been tested so far (giving authors notifications of their accessibility flaws in CHI 2014, having a team of people making fixes on request in CHI 2015). It’s clear that the most effective approach used so far (based on the data in the CACM paper), is a contract-based approach that includes professional support, underwritten by conference or journal budgets. Such an approach will be more effective and consistent than one that relies entirely on author/conference volunteers.
With regard to video accessibility, progress has been very successful. A process for video captioning was piloted with the help of the SIGCHI CMC in 2014/2015, and has now been adopted as part of the SIGCHI Video service: “SIGCHI provides conferences with equipment to record talks… Conferences only pay for shipping the equipment, captioning and one additional student volunteer per track for recording” (emphasis ours). An external company (rev.com) is used to write up the captions, and SIGCHI’s video program chairs ensure they are uploaded with the videos.
Currently, more than 400 videos have been captioned (all keynotes and award talks, all talks from CHI 2015 and 2016). Additionally, this program has been expanded from just CHI to include all upcoming specialized conferences that use SIGCHI video equipment. Conferences added this year include UIST, ISS and SUI.
To summarize, progress has been very successful for videos, but is slow on pdfs, and nothing is known about websites. The SIGCHI Accessibility community believes that the primary barrier to meeting these goals is having enough volunteers to focus on them, and having a commitment to hire professional support to execute them. The success of the videos in comparison to papers is evidence for this. In addition, it is likely that better participation in R1 will be needed before process goals in R2 can be met effectively.
R3: Handle Accessibility Requests
Create a process for handling accessibility requests within SIGCHI1
[Partly met] Create a single point of contact for accessibility questions and advertise it SIGCHI wide. The community has created a single point of contact for support and discussion through the facebook sigchi-access group6. However, there is still not a well known, established mailing list for more formal requests, conference chair support, and so on.
[Not Met] Study the legal context. No effort has been put into this at the moment. However, external work to the SIGCHI Accessibility community has resulted in a first publication on this topic [Kirkham, 2015].
The progress on R3 is very slow. Visibility among the SIGCHI leadership, which has recently increased, should make it easier to establish a communication structure for supporting conferences and we are optimistic that this will change before the next report two years from now. Studies of the legal context is a costly proposition that may be harder to act on, and consideration should be put into what would best lay the groundwork to make this possible.
On the positive side, the SIGCHI Accessibility community has met its long term goal of establishing a position focused on Accessibility among the SIGCHI leadership, specifically with respect to representation on the CHI Steering committee mentioned above. Other long-term goals such as a SIGCHI-wide budget for accessibility have not been met yet.
R4: Increase Representation
Increase representation of people with disabilities within SIGCHI1
[Goal Met] Run a networking event. This goal was met in 2016 in the form of a SIG on accessibility at CHI. In 2017 this was formalized through the addition of accessibility as a component of the Diversity and Inclusion lunch at CHI. Finally, SIGACCESS and the SIGCHI Accessibility community held a joint pizza party at CHI 2017. We hope to see these types of events continue on a yearly basis.
Our short term goal was met for R4, however the larger issue of inclusion is not resolved. It has been gratifying, though the SIGCHI Accessibility community takes no credit, to see the increased use of Beams to address accessibility, as well as the addition of remote committees to the CHI program meeting and the announcement of funding to increase diversity and inclusion events. Both of these help to increase access. However what is not clear is whether there is an increase in participation by people with disabilities or how we would measure that. In addition, there are other long term goals mentioned in the original report that we can consider working on in the next few years, such as fellowships, mentoring, and increased outreach to related stakeholders.
R5: Assess Success
Assess SIGCHI’s success in meeting accessibility guidelines at least once every 2 years1.
[Goal Met] Produce regular reports. This report represents our success on the basic goal of reporting on progress, based on post-conference surveys and other data.
Our immediate goal of assessing and reporting on progress is met by this report. However, we have not established a sustainable approach to collecting the appropriate data on a regular basis, it is still very ad-hoc. Some metrics are ill defined or not tracked.
Assessment of program committees, awards, and so on, has not yet begun. However, we have been working with SIGCHI to ensure that conferences are assessed in a consistent way from year to year. The tentative survey questions we have designed for this purpose are:
Did you request any accessibility-related accommodations for the conference? Yes or No?
If yes, were your requests met? No, a little, some, a lot, Yes
What did you request and how could we improve your experience?
Possible additional questions if the chair wants:
Because we cannot know who did not attend the conference for accessibility reasons, it would be helpful to know about the existence of such issues. Please tell us if you are aware of a situation we should consider, while respecting the privacy of relevant parties.
Finally, the long term goal of taking the burden of reporting off the shoulders of volunteers has not been met. The SIGCHI Accessibility community should consider raising funds to support these efforts over the next few years.
Appendix A: Conference Survey Questions
At conferences, the following questions were asked:
CHI 2014/2015 Did you request any accessibility-related accommodations for the conference? If yes, were your requests met in a timely manner?
CHI 2014-2016 Do you know of any researchers, practitioners, educators or students with disabilities who wanted to attend the CHI conference, but could not because they required specific accommodations? What accommodations could we provide to help them attend?
CSCW 2015 Do you have any suggestions for things the CSCW 2016 planning team should particularly attend to in order to make sure the conference is accessible to people with disabilities?
All years: Sometimes generic questions about the conference received accessibility-spceific responses, which we also analyzed.
Appendix B: Changes to ACM SIGCHI Conferences Policy
ACM strives to make conferences as accessible as possible and provide reasonable accommodations so individuals with disabilities can fully participate. In many countries this is a legal requirement, and compliance should not be taken lightly. We recommend appointing an Accessibility Chair for your conference. SIGACCESS and SIGCHI’s accessibility community have developed guidance for conference organizers that steps you through all stages of conference planning from site selection to the event itself. The SIGCHI accessibility community (reachable at SIGCHIaccess on facebook and sigchi-accessibility@googlegroups.com), SIGACCESS (reachable at chair_sigaccess@acm.org) and ACM support staff stand ready to help you make your event as inclusive as possible.
[Mankoff, 2016] Mankoff, J. (2016). The wicked problem of making SIGCHI accessible. interactions, 23(3), 6-7. DOI 10.1145/2903528
