Color contrast is one such issue. It is of no consequence to people who are totally blind, but it is supremely important to people with partial vision. Still, it affects different people in different ways. Many people who are sensitive to issues of contrast, it seems, need high contrast. But people with low moderate vision, who must use one method or another to magnify the text highly, actually need low contrast—a white background can cause great distress when you must keep your eyes just a few centimeters from the screen so you can read.

That’s just one of many conceivable instances in which a method that improves accessibility for one group of people will create a barrier for others.

So to truly be useful, an assessment of accessibility should define these points:

the task being considered
the nature of the barrier
the severity of the barrier
the type of disability a person must have to be affected by this barrier
perhaps even the type of assistive technology the reviewer assumes the person is using
the skill level of the users affected by the barrier (novice? expert? expert in the subject area, but novice to assistive technology?)
the frequency with which these folks are likely to encounter this barrier

This might seem to be an overwhelming degree of detail, but note that an answer to one question might answer others, as well. For example:

An application that is inaccessible except through the mouse would affect (nearly) all people with visual disabilities or mobility impairments—but would have no impact on people with cognitive disabilities or the Deaf.
A site that requires visitors to find and click a randomly positioned Easter egg before they can get beyond a splash page would be inaccessible to just about everybody. (I hope I didn’t just inspire the hottest new trend among would-be designers.)

This approach would require adding two or three columns to your proposed worksheet. I see several significant advantages:

I would feel less hesitant to publish my assessment of a site’s accessibility, because I could clearly show the limitations of my expertise (or at least of this assessment).
Even if I hadn’t directly considered your situation, you would have greater information about my assessment’s relevance to you.
On more websites, we might see reasonable commitments to continuously improving accessibility rather than unsupported claims that the content is fully accessible.

I’m glad you started this discussion. Great idea!

As a bonus, it’s shorter than the original.

Another audience to consider is older people. The first ever guide to websites for older people (AFAIK) from the Spry Foundation [1] suggests “Fluorescent colors belong in a slightly different category. These colors are supersaturated, but in addition they reflect both normal light and ultraviolet light. The effect is very intense, but can be exhausting to the eye, especially if there are visual disabilities involved. This effect makes fluorescent colors a less than effective choice for text or background.” This is supported by others working with older people [2].

Andrew
[1] http://www.spry.org/pdf/website_creators_guide.pdf
[2] http://otal.umd.edu/UUPractice/elderly/

One such expert warned me that someone could lose their job for not putting in ALT text for an image. What you’ve proposed is sensible and useful.