User Experience Trust Metrics

From IDESG Wiki
Jump to: navigation, search


  • The goal of the NSTIC Strategy is a trusted identifier in cyberspace for every citizen.
  • The goal of the IDESG is creation of ecosystems were user identifiers can be securely used without unnecessary release of User Private Information at a wide range of web sites.
  • The goal of the IDEF requirements is a metric by which a Digital Entity (Relying Party and Provider) can be measured to assure that they are conformant to the IDESG and NSTIC principles.
  • The goal of the IDEF Trustmark is an indicator of trust that will give users confidence in the security and privacy of their interactions with every Digital Entity displaying that mark.
  • The prime directive must be to built frameworks where any Relying Party will find value using the Trustmark and users trust the mark as proof of the trustworthiness of sites displaying the mark.
  • The goal of this paper is to determine whether users' experience the Trustmark meets the design goals, otherwise there will be no value for any Digital Entity to expend the effort to acquire the mark.

The requirements for this page are based on a TFTM presentation of October 2014 and are intended as input to the TFTM process. The following questions are taken from that presentation.

What is the baseline?

Improving the security, privacy, usability, and interoperability of everyday online transactions

What benefits could the everyday consumer see if this baseline were established?

e.g., reduced account compromise through increased use of multifactor authentication; greater user control through notice, consent requirements; etc.

What we do know? (from the NSTIC Strategy Document)

By making online transactions more trustworthy and enhancing consumers' privacy, we will prevent costly crime; we will give businesses and consumers new confidence; and we will foster growth and innovation, online and across our economy — in some ways we can predict, and in other ways we can scarcely imagine. Ultimately, that is the goal of the NSTIC Strategy.


As part of the process of building a IDEF Registry as the first of the IDESG Frameworks, potential users of the site have asked specifically what the benefits of joining such a framework might be.

Our Statement of Trust

The goal of any trust framework is a community of users and providers that will make every person or Digital Entity feel more secure that their privacy, safety and money is not likely to be compromised while operating within the framework.

This is accomplished by:

  1. Verifying that every Digital Entity that is member of the Framework has agreed to follow the principles and rules of the Framework.
  2. Enabling any Digital Entity to be assured that they are communicating with other entities that are currently part of the framework.
  3. Ensuring that users can be authenticated by Identity Providers that will follow the user's consent when release any user Private Information to any other Digital Entity.
  4. Knowing that when a user visits any web site (Relying Party); that site will honor the stipulations that the user places on the entity that holds their User Private Information.

Problem: How to Make IDEF Attractive to Relying Parties

A common adoption problem in most introductory privacy initiatives is they stall when designers of web sites contend with innovative privacy requirements. The biggest challenges are faced by web sites that function as a Relying Party (RP) for identities or attributes provided by Identity or Attribute Providers. In spite of the very real advantage of "owning the customer relationship", these RP web sites have decided to forgo the challenges of securely authenticating users, but they still desire to acquire the most User Private Information possible to further their business interests. In order to enhance user privacy in the ecosystem, trusted IdPs have required user consent to share identity attribute data with RPs, limiting their ability to generate revenue. This creates another challenge: what is the RPs incentive to function within an ecosystem that limits their access to User Private Information? And how do users asses RP requests for more information? Until users can assess the value of an RP as a trusted party that protects a user's privacy and liability, the RP's will have no incentive to comply with any ecosystem rules that both increase their costs and limit that access to User Private Information.

In the private sector, privacy is an individual choice and most of the time it is not valued or appreciated until it is lost or breached, or causes general frustration where some user's don't follow through with building the relationship or reluctantly provide the data because they have little choice. Only where there is a breach or misuse do users personally experience the loss as they try to reestablish their credentials and privacy. Case in point: try rebuilding a credit history once authenticated attributes have been compromised. Users need to educate themselves about privacy certifications, credentials and established third party relationships with a RP supporting functions and this is tedious and not practical. As market forces offer lip service to the importance of privacy and RPs, Governmental efforts to enforce privacy preserving methods and technologies impact the private sector and their relationship with regulated businesses that use Relying Parties. But until both government and market forces work in concert to drive adoption, privacy methods and technology uptake will be limited. Governmental efforts include direct regulatory action and allowing class action lawsuits via a legal process.

The set of metrics provided here apply to how to assess the UXC's requirements (also called Usability Requirements in the documents) as they apply to the IDEF Registry. The IDEF Registry is designed to understand how users embrace trustworthiness and the framework around privacy, security and interoperability, while being resilient when selecting web sites and identity products that serve their interests and meet their objectives on the web for both functionality and security of their money, data they may provide to an entity online, their reputation and their privacy


The following are the points where user experience can be measured to determine if the base line requirements are met. Note that the first metrics are for overall usability moving into the trust measurements that will indicate compliance of a particular implementation with the terms of the IDESG or of the Framework.

How are UX metrics obtained

Metrics are quantitative measures that can be tracked over time. They come from questions presented to user and from the evaluation of direct observations of users on particular sites. The Wikipedia entry on User Experience Evaluation defines these terms: "User experience (UX) evaluation or User experience assessment (UXA) refers to a collection of methods, skills and tools utilized to uncover how a person perceives a system (product, service, non-commercial item, or a combination of them) before, during and after interacting with it. It is non-trivial to assess user experience since user experience is subjective, context-dependent and dynamic over time."

Since the evaluation of the user is dependent on the specific UX presented to the user on a specific site, the broad measure of the success of the whole ecosystem will only be measurable when multiple sites supporting the IDESG ecosystem are widely available and questions about the overall experience are possible. In the meantime, specific implementations of web sites supporting the IDESG are encouraged. The metrics provider here should be able to act as a base set that would allow comparison between different researcher's results. A common base set of metrics would allow the UXC to use independently generated research reports in the compilation of an ecosystem wide report.

(INSERT to flesh out questions) Human Centered Possibility-driven Options-focused Expect to be wrong (portfolio approach) Iterative

All measurements below (expect the verbatim data) should have a qualitative and quantitative ways to measure success from users. Follow up measurements should be included over time.

Measurements (Quantitative)

  1. Can the user accomplish the task set out to accomplish? (A goal might be 90%, a minimum acceptable might be 60%)
  2. What is the System Usability Scale (John Brooke's SUS) <is this helpful in measuring trust? ..tom>
  3. It the Trustmark discoverable and self-describing. (90%, 70%)
  4. Does the user feel safer as a result of the appearance of the Trustmark. (99%, 80% of those answering in the affirmative above.)
  5. Does the user feel that the site is safe overall? (the metric is a comparisons of the positives to the negatives.)
  6. Does the user understand the necessity for a strong identity in protecting their money, their reputation and their privacy?
  7. Does the user know whether the identity of the provider is strongly bound to a real-world, trusted entity?
  8. Collected verbatim responses used for site improvement.

Open question, does there need to be a separate list for a Relying Party and for an Identity Provider.

The System Usability Scale (Survey)

The SUS is a 10 item questionnaire with 5 response options.

  1. I think that I would like to use this framework frequently.
  2. I found the framework unnecessarily complex.
  3. I thought the sites that adhere to the framework are easy to use.
  4. I think that I would need the support of a technical person to be able to use this framework.
  5. I found the various functions in this web site were well integrated.
  6. I thought there was too much inconsistency in this web site.
  7. I would imagine that most people would learn to use this framework very quickly.
  8. I found the framework was more trouble than it was worth.
  9. I felt very confident using web sites that display the framework logo.
  10. I needed to learn a lot of things before I could get going with this framework.

Guidelines of Behavior for Designing Surveys and Evaluating Results

Usability researchers should adhere to IRB requirements for the treatment of users and user collected information.

Survey guidance
1. Consent to participate must be voluntarily given, as well as the option to decline participation in the survey.
2. Individuals should be able to exit the survey at any time.
3. Survey questions should be comprehendible by a wide audience.
4. Surveys should be concise and easy to answer.
5. Surveys should be open to all users to ensure the widest possible range of individual respondents.
6. Surveys should include an introductory explanation of how the user's personal information and responses will be treated, what level confidentiality they can expect and links to privacy policy, trust frameworks and other policies that govern collection of personal information.
7. Any results of surveys should be aggregated and depersonalized to protect the participant's privacy.
8. Results of the survey could be made available as long as privacy of participants can be insured by the system.

User Interviews (Qualitative Research and Ethnographic Studies)

Qualitative Research generally involves direct in-person interviews with human subjects, and as such requires care with respect to personal information and user stories shared. NSTIC pilots are required to use formal academic IRB standards and certification, and generally erring on the side of personal information privacy and confidentiality is preferred when conducting direct interviews with subjects or collecting other personal information.

Qualitative Research is conducted in order to gather in-depth information about the reasons subjects choose one path or method over another, understand systems one way verses another, etc. It is conducted in an effort to get at the deeper issues in a system, that cannot be understood just by reviewing usage logs or via survey data which rarely explains why or how people understand a system. Additionally, often subjects develop work-arounds for systems that don't work, and it is through watching them work that the original problem the work-around is meant to solve becomes apparent. It is this understanding that often isn't quite clear to the subject that is often discovered through interviews.

Guidelines of Behavior for Designing Qualitative Research

Qualitative Research can be conducted in person, through journal entry, via group discussion and observations in personal settings. Due to the personal nature of this research, some guidelines are included below:

Robert Wood Johnson Foundation Qualitative Research Guidelines: and Evaluative Criteria
Cochrane Qualitative and Implementation Methods Group

References and Coordination