User Experience Trust Metrics - Revision history

Mary Hodder: /* Problem: How to Make IDEF Attractive to Relying Parties */ small syntax cleanup

2017-09-28T19:04:04Z

Problem: How to Make IDEF Attractive to Relying Parties: small syntax cleanup

New page

==Introduction==
User Research should enable measurement of the evolving trust that is enabled by the work of the IDESG and the IDEF registry.

The requirements for this page are based on a TFTM presentation of October 2014 and are intended as input to the TFTM process. The following questions are taken from that presentation.
===What is the baseline?===
Improving the security, privacy, usability, and interoperability of everyday online transactions
===What benefits could the everyday consumer see if this baseline were established?===
e.g., reduced account compromise through increased use of multifactor authentication; greater user control through notice, consent requirements; etc.

===What we do know? (from the [[NSTIC Strategy]] Document)===
By making online transactions more trustworthy and enhancing consumers' privacy, we
will prevent costly crime; we will give businesses and consumers new confidence; and we will foster
growth and innovation, online and across our economy — in some ways we can predict, and in other
ways we can scarcely imagine. Ultimately, that is the goal of the [[NSTIC Strategy]].

==Context==
As part of the process of building a IDEF Registry as the first of the IDESG Frameworks, potential users of the site have asked specifically what the benefits of joining such a framework might be.

===Our Statement of Trust===
The goal of any trust framework is a community of users and providers that will make every person or [[Digital Entity]] feel more secure that their privacy, safety and money is not likely to be compromised while operating within the framework.

This is accomplished by:
#Verifying that every [[Digital Entity]] that is member of the Framework has agreed to follow the principles and rules of the Framework.
#Enabling any [[Digital Entity]] to be assured that they are communicating with other entities that are currently part of the framework.
#Ensuring that users can be authenticated by Identity Providers that will follow the user's consent when release any [[user Private Information]] to any other [[Digital Entity]].
#Knowing that when a user visits any web site ([[Relying Party]]); that site will honor the stipulations that the user places on the entity that holds their [[User Private Information]].

==Problem: How to Make IDEF Attractive to Relying Parties==

A common uptake problem in most introductory privacy initiatives is they stall when designers of web sites contend with innovative privacy requirements. The biggest challenges are faced by web sites that function as a [[Relying Party]] (RP) for identities or attributes provided by Identity or Attribute Providers. In spite of the very real advantage of "owning the customer relationship", these RP web sites have decided to forgo the challenges of securely authenticating users, but they still desire to acquire the most [[User Private Information]] possible to further their business interests. In order to enhance user privacy in the ecosystem, trusted IdPs have required user consent to share identity attribute data with RPs, limiting their ability to generate revenue. This creates another challenge: what is the RPs incentive to function within an ecosystem that limits their access to [[User Private Information]]? And how do users asses RP requests for more information? Until users can assess the value of an RP as a trusted party that protects a user's privacy and liability, the RP's will have no incentive to comply with any ecosystem rules that both increase their costs and limit that access to [[User Private Information]].

In the private sector, privacy is an individual choice and most of the time it is not valued or appreciated until it is lost or breached, or causes general frustration where some user's don't follow through with building the relationship or reluctantly provide the data because they have little choice. Only where there is a breach or misuse do users personally experience the loss as they try to reestablish their credentials and privacy. Case in point: try rebuilding a credit history once authenticated attributes have been compromised. Users need to educate themselves about privacy certifications, credentials and established third party relationships with a RP supporting functions and this is tedious and not practical. As market forces offer lip service to the importance of privacy and RPs, Governmental efforts to enforce privacy preserving methods and technologies impact the private sector and their relationship with regulated businesses that use Relying Parties. But until both government and market forces work in concert to drive adoption, privacy methods and technology uptake will be limited. Governmental efforts include direct regulatory action and allowing class action lawsuits via a legal process.

The set of metrics provided here apply to how to assess the UXC's requirements (also called Usability Requirements in the documents) as they apply to the [http://idefregistry.org IDEF Registry]. The IDEF Registry is designed to understand how users embrace trustworthiness and the framework around privacy, security and interoperability, while being resilient when selecting web sites and identity products that serve their interests and meet their objectives on the web for both functionality and security of their money, data they may provide to an entity online, their reputation and their privacy

==Metrics==
The following are the points where user experience can be measured to determine if the base line requirements are met. Note that the first metrics are for overall usability moving into the trust measurements that will indicate compliance of a particular implementation with the terms of the IDESG or of the Framework.
===How are UX metrics obtained===
Metrics are quantitative measures that can be tracked over time. They come from questions presented to user and from the evaluation of direct observations of users on particular sites. The Wikipedia entry on User Experience Evaluation (at this site [[http://en.wikipedia.org/wiki/User_experience_evaluation]]) defines these terms: "'''User experience (UX) evaluation''' or '''User experience assessment (UXA)''' refers to a collection of methods, skills and tools utilized to uncover how a person perceives a system (product, service, non-commercial item, or a combination of them) before, during and after interacting with it. It is non-trivial to assess user experience since user experience is subjective, context-dependent and dynamic over time."

Since the evaluation of the user is dependent on the specific UX presented to the user on a specific site, the broad measure of the success of the whole ecosystem will only be measurable when multiple sites supporting the IDESG ecosystem are widely available and questions about the overall experience are possible. In the meantime, specific implementations of web sites supporting the IDESG are encouraged. The metrics provider here should be able to act as a base set that would allow comparison between different researcher's results. A common base set of metrics would allow the UXC to use independently generated research reports in the compilation of an ecosystem wide report.

(INSERT to flesh out questions)
Human Centered
Possibility-driven
Options-focused
Expect to be wrong (portfolio approach)
Iterative

All measurements below (expect the verbatim data) should have a qualitative and quantitative ways to measure success from users. Follow up measurements should be included over time.

===Measurements (Quantitative)===
# Can the user accomplish the task set out to accomplish? (A goal might be 90%, a minimum acceptable might be 60%)
# What is the System Usability Scale (John Brooke's SUS)
# It the Trustmark discoverable and self-describing. (90%, 70%)
# Does the user feel safer as a result of the appearance of the Trustmark. (99%, 80% of those answering in the affirmative above.)
# Does the user feel that the site is safe overall? (the metric is a comparisons of the positives to the negatives.)
# Does the user understand the necessity for a strong identity in protecting their money, their reputation and their privacy?
# Does the user know whether the identity of the provider is strongly bound to a real-world, trusted entity?
# Collected verbatim responses used for site improvement.

Open question, does there need to be a separate list for a [[Relying Party]] and for an [[Identity Provider]].

===The System Usability Scale (Survey)===
The SUS is a 10 item questionnaire with 5 response options.
#I think that I would like to use this framework frequently.
#I found the framework unnecessarily complex.
#I thought the sites that adhere to the framework are easy to use.
#I think that I would need the support of a technical person to be able to use this framework.
#I found the various functions in this web site were well integrated.
#I thought there was too much inconsistency in this web site.
#I would imagine that most people would learn to use this framework very quickly.
#I found the framework was more trouble than it was worth.
#I felt very confident using web sites that display the framework logo.
#I needed to learn a lot of things before I could get going with this framework.

====Guidelines of Behavior for Designing Surveys and Evaluating Results====
Usability researchers should adhere to IRB requirements for the treatment of users and user collected information.

Survey guidance 
1. Consent to participate must be voluntarily given, as well as the option to decline participation in the survey. 
2. Individuals should be able to exit the survey at any time. 
3. Survey questions should be comprehendible by a wide audience. 
4. Surveys should be concise and easy to answer. 
5. Surveys should be open to all users to ensure the widest possible range of individual respondents. 
6. Surveys should include an introductory explanation of how the user's personal information and responses will be treated, what level confidentiality they can expect and links to privacy policy, trust frameworks and other policies that govern collection of personal information. 
7. Any results of surveys should be aggregated and depersonalized to protect the participant's privacy. 
8. Results of the survey could be made available as long as privacy of participants can be insured by the system. 

===User Interviews (Qualitative Research and Ethnographic Studies)===

Qualitative Research generally involves direct in-person interviews with human subjects, and as such requires care with respect to personal information and user stories shared. NSTIC pilots are required to use formal academic IRB standards and certification, and generally erring on the side of personal information privacy and confidentiality is preferred when conducting direct interviews with subjects or collecting other personal information.

Qualitative Research is conducted in order to gather in-depth information about the reasons subjects choose one path or method over another, understand systems one way verses another, etc. It is conducted in an effort to get at the deeper issues in a system, that cannot be understood just by reviewing usage logs or via survey data which rarely explains why or how people understand a system. Additionally, often subjects develop work-arounds for systems that don't work, and it is through watching them work that the original problem the work-around is meant to solve becomes apparent. It is this understanding that often isn't quite clear to the subject that is often discovered through interviews.

====Guidelines of Behavior for Designing Qualitative Research====

Qualitative Research can be conducted in person, through journal entry, via group discussion and observations in personal settings. Due to the personal nature of this research, some guidelines are included below:

Robert Wood Johnson Foundation Qualitative Research Guidelines: http://www.qualres.org/ and Evaluative Criteria http://www.qualres.org/HomeEval-3664.html 
Cochrane Qualitative and Implementation Methods Group http://cqim.cochrane.org/ 

==References and Coordination==
#http://www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/other-resources-links
#http://www.section508.gov/best-practices

[[Category:User Experience]]
[[Category:Trust]]

Mary Hodder: /* Problem */ rewrite

2017-09-26T17:04:00Z

Problem: rewrite

New page

==Introduction==
User Research should enable measurement of the evolving trust that is enabled by the work of the IDESG and the IDEF registry.

The requirements for this page are based on a TFTM presentation of October 2014 and are intended as input to the TFTM process. The following questions are taken from that presentation.
===What is the baseline?===
Improving the security, privacy, usability, and interoperability of everyday online transactions
===What benefits could the everyday consumer see if this baseline were established?===
e.g., reduced account compromise through increased use of multifactor authentication; greater user control through notice, consent requirements; etc.

===What we do know? (from the [[NSTIC Strategy]] Document)===
By making online transactions more trustworthy and enhancing consumers' privacy, we
will prevent costly crime; we will give businesses and consumers new confidence; and we will foster
growth and innovation, online and across our economy — in some ways we can predict, and in other
ways we can scarcely imagine. Ultimately, that is the goal of the [[NSTIC Strategy]].

==Context==
As part of the process of building a IDEF Registry as the first of the IDESG Frameworks, potential users of the site have asked specifically what the benefits of joining such a framework might be.

===Our Statement of Trust===
The goal of any trust framework is a community of users and providers that will make every person or [[Digital Entity]] feel more secure that their privacy, safety and money is not likely to be compromised while operating within the framework.

This is accomplished by:
#Verifying that every [[Digital Entity]] that is member of the Framework has agreed to follow the principles and rules of the Framework.
#Enabling any [[Digital Entity]] to be assured that they are communicating with other entities that are currently part of the framework.
#Ensuring that users can be authenticated by Identity Providers that will follow the user's consent when release any [[user Private Information]] to any other [[Digital Entity]].
#Knowing that when a user visits any web site ([[Relying Party]]); that site will honor the stipulations that the user places on the entity that holds their [[User Private Information]].

==Problem: How to Make IDEF Attractive to Relying Parties==

The uptake of any privacy initiative has always stalled at uptake of the privacy requirements by the [[Relying Party]]. The [[Relying Party]] needs to acquire customers and other users to attract business or ad revenue. Traditionally this has meant that either many Identity Providers (IDP) didn't want to also be Relying Parties, IE take other IDPs identities, and that entities wanted to show business growth and use via their "user numbers." But in order that individuals gain more privacy in the system, through maintaining their identity information at a trusted IDP and then providing limited data to RPs, RPs have to be willing to take less information from individuals. What is the RPs incentive to function strictly as an RP?

Until the customers and user decide that privacy is important to them, the [[Relying Party]] will always view privacy initiatives as a cost that has no benefit to them. And until RPs decide that having less data is better, because they hold less than can be hacked and stolen, and therefore have less liability while still being able to run their businesses effectively selling their real products (not individual's data), there will be fewer RPs. Additionally, before individuals both value privacy and have a choice that includes web sites that honor privacy, there will be no market forces to drive privacy, because they will not understand how to create more privacy and how to put pressure in markets to support RPs by providing them the individual's business.

Governmental efforts to enforce privacy preserving methods and technologies will have some impact on [[Relying Party]] adoption, but until both government and market forces work in concert to drive adoption, privacy methods and technology uptake will be limited. Governmental efforts include direct regulatory action and supporting class action lawsuits via the legal process.

The set of metrics provided here apply to how to asses the UXC's requirements as they apply to the [IDEF Registry|http://idefregistry.org]. The IDEF Registry is designed to understand how users see privacy, security and interoperability when selecting web sites and identity products that are most likely to serve their interests and meet their objectives on the web for both functionality and security of their money, data they may provide to an entity online, their reputation and their privacy.

==Metrics==
The following are the points where user experience can be measured to determine if the base line requirements are met. Note that the first metrics are for overall usability moving into the trust measurements that will indicate compliance of a particular implementation with the terms of the IDESG or of the Framework.
===How are UX metrics obtained===
Metrics are quantitative measures that can be tracked over time. They come from questions presented to user and from the evaluation of direct observations of users on particular sites. The Wikipedia entry on User Experience Evaluation (at this site [[http://en.wikipedia.org/wiki/User_experience_evaluation]]) defines these terms: "'''User experience (UX) evaluation''' or '''User experience assessment (UXA)''' refers to a collection of methods, skills and tools utilized to uncover how a person perceives a system (product, service, non-commercial item, or a combination of them) before, during and after interacting with it. It is non-trivial to assess user experience since user experience is subjective, context-dependent and dynamic over time."

Since the evaluation of the user is dependent on the specific UX presented to the user on a specific site, the broad measure of the success of the whole ecosystem will only be measurable when multiple sites supporting the IDESG ecosystem are widely available and questions about the overall experience are possible. In the meantime, specific implementations of web sites supporting the IDESG are encouraged. The metrics provider here should be able to act as a base set that would allow comparison between different researcher's results. A common base set of metrics would allow the UXC to use independently generated research reports in the compilation of an ecosystem wide report.

(INSERT to flesh out questions)
Human Centered
Possibility-driven
Options-focused
Expect to be wrong (portfolio approach)
Iterative

All measurements below (expect the verbatim data) should have a qualitative and quantitative ways to measure success from users. Follow up measurements should be included over time.

===Measurements (Quantitative)===
# Can the user accomplish the task set out to accomplish? (A goal might be 90%, a minimum acceptable might be 60%)
# What is the System Usability Scale (John Brooke's SUS)
# It the Trustmark discoverable and self-describing. (90%, 70%)
# Does the user feel safer as a result of the appearance of the Trustmark. (99%, 80% of those answering in the affirmative above.)
# Does the user feel that the site is safe overall? (the metric is a comparisons of the positives to the negatives.)
# Does the user understand the necessity for a strong identity in protecting their money, their reputation and their privacy?
# Does the user know whether the identity of the provider is strongly bound to a real-world, trusted entity?
# Collected verbatim responses used for site improvement.

Open question, does there need to be a separate list for a [[Relying Party]] and for an [[Identity Provider]].

===The System Usability Scale (Survey)===
The SUS is a 10 item questionnaire with 5 response options.
#I think that I would like to use this framework frequently.
#I found the framework unnecessarily complex.
#I thought the sites that adhere to the framework are easy to use.
#I think that I would need the support of a technical person to be able to use this framework.
#I found the various functions in this web site were well integrated.
#I thought there was too much inconsistency in this web site.
#I would imagine that most people would learn to use this framework very quickly.
#I found the framework was more trouble than it was worth.
#I felt very confident using web sites that display the framework logo.
#I needed to learn a lot of things before I could get going with this framework.

====Guidelines of Behavior for Designing Surveys and Evaluating Results====
Usability researchers should adhere to IRB requirements for the treatment of users and user collected information.

Survey guidance 
1. Consent to participate must be voluntarily given, as well as the option to decline participation in the survey. 
2. Individuals should be able to exit the survey at any time. 
3. Survey questions should be comprehendible by a wide audience. 
4. Surveys should be concise and easy to answer. 
5. Surveys should be open to all users to ensure the widest possible range of individual respondents. 
6. Surveys should include an introductory explanation of how the user's personal information and responses will be treated, what level confidentiality they can expect and links to privacy policy, trust frameworks and other policies that govern collection of personal information. 
7. Any results of surveys should be aggregated and depersonalized to protect the participant's privacy. 
8. Results of the survey could be made available as long as privacy of participants can be insured by the system. 

===User Interviews (Qualitative Research and Ethnographic Studies)===

Qualitative Research generally involves direct in-person interviews with human subjects, and as such requires care with respect to personal information and user stories shared. NSTIC pilots are required to use formal academic IRB standards and certification, and generally erring on the side of personal information privacy and confidentiality is preferred when conducting direct interviews with subjects or collecting other personal information.

Qualitative Research is conducted in order to gather in-depth information about the reasons subjects choose one path or method over another, understand systems one way verses another, etc. It is conducted in an effort to get at the deeper issues in a system, that cannot be understood just by reviewing usage logs or via survey data which rarely explains why or how people understand a system. Additionally, often subjects develop work-arounds for systems that don't work, and it is through watching them work that the original problem the work-around is meant to solve becomes apparent. It is this understanding that often isn't quite clear to the subject that is often discovered through interviews.

====Guidelines of Behavior for Designing Qualitative Research====

Qualitative Research can be conducted in person, through journal entry, via group discussion and observations in personal settings. Due to the personal nature of this research, some guidelines are included below:

Robert Wood Johnson Foundation Qualitative Research Guidelines: http://www.qualres.org/ and Evaluative Criteria http://www.qualres.org/HomeEval-3664.html 
Cochrane Qualitative and Implementation Methods Group http://cqim.cochrane.org/ 

==References and Coordination==
#http://www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/other-resources-links
#http://www.section508.gov/best-practices

[[Category:User Experience]]
[[Category:Trust]]

Mary Hodder: /* Guidelines of Behavior for Designing Surveys and Evaluating Results */

2015-03-05T21:10:01Z

Guidelines of Behavior for Designing Surveys and Evaluating Results

← Older revision		Revision as of 21:10, 5 March 2015
Line 50:		Line 50:
	Usability researchers should adhere to IRB requirements for the treatment of users and user collected information.		Usability researchers should adhere to IRB requirements for the treatment of users and user collected information.

	Survey guidance		Survey guidance <br>
	1. Consent to participate must be voluntarily given, as well as the option to decline participation in the survey.		1. Consent to participate must be voluntarily given, as well as the option to decline participation in the survey. <br>
	2. Individuals should be able to exit the survey at any time.		2. Individuals should be able to exit the survey at any time. <br>
	3. Survey questions should be comprehendible by a wide audience.		3. Survey questions should be comprehendible by a wide audience.<br>
	4. Surveys should be concise and easy to answer.		4. Surveys should be concise and easy to answer. <br>
	5. Surveys should be open to all users to ensure the widest possible range of individual respondents.		5. Surveys should be open to all users to ensure the widest possible range of individual respondents. <br>
	6. Surveys should include an introductory explanation of how the user's personal information and responses will be treated, what level confidentiality they can expect and links to privacy policy, trust frameworks and other policies that govern collection of personal information.		6. Surveys should include an introductory explanation of how the user's personal information and responses will be treated, what level confidentiality they can expect and links to privacy policy, trust frameworks and other policies that govern collection of personal information. <br>
	7. Any results of surveys should be aggregated and depersonalized to protect the participant's privacy.		7. Any results of surveys should be aggregated and depersonalized to protect the participant's privacy. <br>
	8. Results of the survey could be made available as long as privacy of participants can be insured by the system.		8. Results of the survey could be made available as long as privacy of participants can be insured by the system.<br>

	===User Interviews (Qualitative Research and Ethnographic Studies)===		===User Interviews (Qualitative Research and Ethnographic Studies)===

Mary Hodder: /* Guidelines of Behavior for Designing Surveys and Evaluating Results */ Added survey guidance

2015-03-03T17:54:18Z

Guidelines of Behavior for Designing Surveys and Evaluating Results: Added survey guidance

← Older revision		Revision as of 17:54, 3 March 2015
Line 48:		Line 48:

	====Guidelines of Behavior for Designing Surveys and Evaluating Results====		====Guidelines of Behavior for Designing Surveys and Evaluating Results====
	Usability researchers should adhere to IRB requirements for the treatment of users and user collected information.		Usability researchers should adhere to IRB requirements for the treatment of users and user collected information.

			Survey guidance
			1. Consent to participate must be voluntarily given, as well as the option to decline participation in the survey.
			2. Individuals should be able to exit the survey at any time.
			3. Survey questions should be comprehendible by a wide audience.
			4. Surveys should be concise and easy to answer.
			5. Surveys should be open to all users to ensure the widest possible range of individual respondents.
			6. Surveys should include an introductory explanation of how the user's personal information and responses will be treated, what level confidentiality they can expect and links to privacy policy, trust frameworks and other policies that govern collection of personal information.
			7. Any results of surveys should be aggregated and depersonalized to protect the participant's privacy.
			8. Results of the survey could be made available as long as privacy of participants can be insured by the system.

	===User Interviews (Qualitative Research and Ethnographic Studies)===		===User Interviews (Qualitative Research and Ethnographic Studies)===

Mary Hodder: /* Guidelines of Behavior for Designing Qualitative Research */

2015-03-03T15:32:41Z

Guidelines of Behavior for Designing Qualitative Research

← Older revision		Revision as of 15:32, 3 March 2015
Line 63:		Line 63:
	Cochrane Qualitative and Implementation Methods Group http://cqim.cochrane.org/ <br>		Cochrane Qualitative and Implementation Methods Group http://cqim.cochrane.org/ <br>

	~~[[Category~~:~~User Experience]]~~		Additional links to help research:<br>
	~~[[Category:Concept]]~~


	~~Placeholder links~~
	http://www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/other-resources-links <br>		http://www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/other-resources-links <br>
	http://abledata.com/abledata.cfm?CFID=83403419&CFTOKEN=2eec2aeffb602fc3-C88E7ADB-9730-C534-AC45C274BDDACBCC <Br>		http://abledata.com/abledata.cfm?CFID=83403419&CFTOKEN=2eec2aeffb602fc3-C88E7ADB-9730-C534-AC45C274BDDACBCC <Br>
	http://abledata.com/abledata.cfm?pageid=113709&ksectionid=19327 <br>		http://abledata.com/abledata.cfm?pageid=113709&ksectionid=19327 <br>
	http://www.section508.gov/best-practices <br>		http://www.section508.gov/best-practices <br>

			[[Category:User Experience]]
			[[Category:Concept]]

Mary Hodder: /* User Interviews (Qualitative Research and Ethnographic Studies) */

2015-03-03T15:28:46Z

User Interviews (Qualitative Research and Ethnographic Studies)

← Older revision		Revision as of 15:28, 3 March 2015
Line 52:		Line 52:
	===User Interviews (Qualitative Research and Ethnographic Studies)===		===User Interviews (Qualitative Research and Ethnographic Studies)===

	Qualitative Research generally ~~involved~~ direct in-person interviews with human subjects, and as such requires care with respect to personal information and user ~~story~~ shared. NSTIC pilots are required to use formal academic IRB standards and certification, and generally erring on the side of personal information privacy and confidentiality is preferred when conducting direct interviews with subjects.		Qualitative Research generally involves direct in-person interviews with human subjects, and as such requires care with respect to personal information and user stories shared. NSTIC pilots are required to use formal academic IRB standards and certification, and generally erring on the side of personal information privacy and confidentiality is preferred when conducting direct interviews with subjects or collecting other personal information.

	Qualitative Research is conducted in order to gather in-depth information about the reasons subjects choose one path or method over another, understand systems one way verses another, etc. It is conducted in an effort to get at the deeper issues in a system, that cannot be understood just by reviewing usage logs or via survey data which rarely explains why or how people understand a system. Additionally, often subjects develop work-arounds for systems that don't work, and it is through watching them work that the original problem the work-around is meant to solve becomes apparent. It is this understanding that often isn't quite clear to the subject that is often discovered through interviews.		Qualitative Research is conducted in order to gather in-depth information about the reasons subjects choose one path or method over another, understand systems one way verses another, etc. It is conducted in an effort to get at the deeper issues in a system, that cannot be understood just by reviewing usage logs or via survey data which rarely explains why or how people understand a system. Additionally, often subjects develop work-arounds for systems that don't work, and it is through watching them work that the original problem the work-around is meant to solve becomes apparent. It is this understanding that often isn't quite clear to the subject that is often discovered through interviews.

Mary Hodder: /* Guidelines of Behavior for Designing Qualitative Research */

2015-03-03T15:26:54Z

Guidelines of Behavior for Designing Qualitative Research

← Older revision		Revision as of 15:26, 3 March 2015
Line 60:		Line 60:
	Qualitative Research can be conducted in person, through journal entry, via group discussion and observations in personal settings. Due to the personal nature of this research, some guidelines are included below:		Qualitative Research can be conducted in person, through journal entry, via group discussion and observations in personal settings. Due to the personal nature of this research, some guidelines are included below:

	Robert Wood Johnson Foundation Evaluative Criteria http://www.qualres.org/HomeEval-3664.html <br>		Robert Wood Johnson Foundation Qualitative Research Guidelines: http://www.qualres.org/ and Evaluative Criteria http://www.qualres.org/HomeEval-3664.html <br>
	Cochrane Qualitative and Implementation Methods Group http://cqim.cochrane.org/ <br>		Cochrane Qualitative and Implementation Methods Group http://cqim.cochrane.org/ <br>

Mary Hodder: /* User Interviews (Qualitative Research and Ethnographic Studies) */ added explanation of Qualitative Research and links to guidelines for QualR.

2015-03-03T15:23:48Z

User Interviews (Qualitative Research and Ethnographic Studies): added explanation of Qualitative Research and links to guidelines for QualR.

← Older revision		Revision as of 15:23, 3 March 2015
Line 51:		Line 51:

	===User Interviews (Qualitative Research and Ethnographic Studies)===		===User Interviews (Qualitative Research and Ethnographic Studies)===

			Qualitative Research generally involved direct in-person interviews with human subjects, and as such requires care with respect to personal information and user story shared. NSTIC pilots are required to use formal academic IRB standards and certification, and generally erring on the side of personal information privacy and confidentiality is preferred when conducting direct interviews with subjects.

			Qualitative Research is conducted in order to gather in-depth information about the reasons subjects choose one path or method over another, understand systems one way verses another, etc. It is conducted in an effort to get at the deeper issues in a system, that cannot be understood just by reviewing usage logs or via survey data which rarely explains why or how people understand a system. Additionally, often subjects develop work-arounds for systems that don't work, and it is through watching them work that the original problem the work-around is meant to solve becomes apparent. It is this understanding that often isn't quite clear to the subject that is often discovered through interviews.

	====Guidelines of Behavior for Designing Qualitative Research====		====Guidelines of Behavior for Designing Qualitative Research====

			Qualitative Research can be conducted in person, through journal entry, via group discussion and observations in personal settings. Due to the personal nature of this research, some guidelines are included below:

			Robert Wood Johnson Foundation Evaluative Criteria http://www.qualres.org/HomeEval-3664.html <br>
			Cochrane Qualitative and Implementation Methods Group http://cqim.cochrane.org/ <br>

	[[Category:User Experience]]		[[Category:User Experience]]
Line 59:		Line 68:

	Placeholder links		Placeholder links
	http://www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/other-resources-links		http://www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/other-resources-links <br>
	http://abledata.com/abledata.cfm?CFID=83403419&CFTOKEN=2eec2aeffb602fc3-C88E7ADB-9730-C534-AC45C274BDDACBCC		http://abledata.com/abledata.cfm?CFID=83403419&CFTOKEN=2eec2aeffb602fc3-C88E7ADB-9730-C534-AC45C274BDDACBCC <Br>
	http://abledata.com/abledata.cfm?pageid=113709&ksectionid=19327		http://abledata.com/abledata.cfm?pageid=113709&ksectionid=19327 <br>
	http://www.section508.gov/best-practices		http://www.section508.gov/best-practices <br>

Mary Hodder: UXC requirements working meeting - changes

2015-02-26T22:40:59Z

UXC requirements working meeting - changes

New page

==Introduction==
UX Metrics should enable measurement of the evolving baseline for participation in the Identity Ecosystem.

The requirements for this page are based on a TFTM presentation of October 2014 and are intended as input to the TFTM process. The following questions are taken from that presentation.
===What is the baseline?===
Improving the security, privacy, usability, and interoperability of everyday online transactions
===What benefits could the everyday consumer see if this baseline was established?===
e.g., reduced account compromise through increased use of multifactor authentication; greater user control through notice, consent requirements; etc.

==Content==
The following are the points where user experience can be measured to determine if the base line requirements are met. Note that the first metrics are for overall usability moving into the trust measurements that will indicate compliance of a particular implementation with the terms of the IDESG or of the Framework.
===How are UX metrics obtained===
Metrics are quantitative measures that can be tracked over time. They come from questions presented to user and from the evaluation of direct observations of users on particular sites. The Wikipedia entry on User Experience Evaluation (at this site [[http://en.wikipedia.org/wiki/User_experience_evaluation]]) defines these terms: "'''User experience (UX) evaluation''' or '''User experience assessment (UXA)''' refers to a collection of methods, skills and tools utilized to uncover how a person perceives a system (product, service, non-commercial item, or a combination of them) before, during and after interacting with it. It is non-trivial to assess user experience since user experience is subjective, context-dependent and dynamic over time."

Since the evaluation of the user is dependent on the specific UX presented to the user on a specific site, the broad measure of the success of the whole ecosystem will only be measurable when multiple sites supporting the IDESG ecosystem are widely available and questions about the overall experience are possible. In the meantime, specific implementations of web sites supporting the IDESG are encouraged. The metrics provider here should be able to act as a base set that would allow comparison between different researcher's results. A common base set of metrics would allow the UXC to use independently generated research reports in the compilation of an ecosystem wide report.

(INSERT to flesh out questions)
Human Centered
Possibility-driven
Options-focused
Expect to be wrong (portfolio approach)
Iterative

{{All measurements below (expect the verbatim data) should have a qualitative and quantitative ways to measure success from users. Follow up measurements should be included over time.}}

===Measurements (Quantitative)===
# Can the user accomplish the task set out to accomplish? (A goal might be 90%, a minimum acceptable might be 60%)
# What is the System Usability Scale (John Brooke's SUS)
# It the Trustmark discoverable and self-describing. (90%, 70%)
# Does the user feel safer as a result of the appearance of the Trustmark. (99%, 80% of those answering in the affirmative above.)
# Does the user feel that the site is safe overall? (the metric is a comparisons of the positives to the negatives.)
# Does the user understand the necessity for a strong identity for their providers?
# Does the user know whether the identity of the provider is strongly bound to a real-world entity?
# Collected verbatim responses used for site improvement.

===The System Usability Scale (Survey)===
The SUS is a 10 item questionnaire with 5 response options.
#I think that I would like to use this system frequently.
#I found the system unnecessarily complex.
#I thought the system was easy to use.
#I think that I would need the support of a technical person to be able to use this system.
#I found the various functions in this system were well integrated.
#I thought there was too much inconsistency in this system.
#I would imagine that most people would learn to use this system very quickly.
#I found the system very cumbersome to use.
#I felt very confident using the system.
#I needed to learn a lot of things before I could get going with this system.

====Guidelines of Behavior for Designing Surveys and Evaluating Results====
Usability researchers should adhere to IRB requirements for the treatment of users and user collected information.

===User Interviews (Qualitative Research and Ethnographic Studies)===

====Guidelines of Behavior for Designing Qualitative Research====

[[Category:User Experience]]
[[Category:Concept]]

Placeholder links
http://www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/other-resources-links
http://abledata.com/abledata.cfm?CFID=83403419&CFTOKEN=2eec2aeffb602fc3-C88E7ADB-9730-C534-AC45C274BDDACBCC
http://abledata.com/abledata.cfm?pageid=113709&ksectionid=19327
http://www.section508.gov/best-practices

Mary Hodder: Added potential metrics requirement

2015-01-27T18:01:28Z

Added potential metrics requirement

New page

==Introduction==
UX Metrics should enable measurement of the evolving baseline for participation in the Identity Ecosystem.

The requirements for this page are based on a TFTM presentation of October 2014 and are intended as input to the TFTM process. The following questions are taken from that presentation.
===What is the baseline?===
Improving the security, privacy, usability, and interoperability of everyday online transactions
===What benefits could the everyday consumer see if this baseline was established?===
e.g., reduced account compromise through increased use of multifactor authentication; greater user control through notice, consent requirements; etc.

==Content==
The following are the points where user experience can be measured to determine if the base line requirements are met. Note that the first metrics are for overall usability moving into the trust measurements that will indicate compliance of a particular implementation with the terms of the IDESG or of the Framework.
===How are UX metrics obtained===
Metrics are quantitative measures that can be tracked over time. They come from questions presented to user and from the evaluation of direct observations of users on particular sites. The Wikipedia entry on User Experience Evaluation (at this site [[http://en.wikipedia.org/wiki/User_experience_evaluation]]) defines these terms: "'''User experience (UX) evaluation''' or '''User experience assessment (UXA)''' refers to a collection of methods, skills and tools utilized to uncover how a person perceives a system (product, service, non-commercial item, or a combination of them) before, during and after interacting with it. It is non-trivial to assess user experience since user experience is subjective, context-dependent and dynamic over time."

Since the evaluation of the user is dependent on the specific UX presented to the user on a specific site, the broad measure of the success of the whole ecosystem will only be measurable when multiple sites supporting the IDESG ecosystem are widely available and questions about the overall experience are possible. In the meantime, specific implementations of web sites supporting the IDESG are encouraged. The metrics provider here should be able to act as a base set that would allow comparison between different researcher's results. A common base set of metrics would allow the UXC to use independently generated research reports in the compilation of an ecosystem wide report.

{{All measurements below should have a qualitative and quantitative ways to measure success from users. Follow up measurements should be included over time.}}

===Measurements===
# Can the user accomplish the task set out to accomplish? (A goal might be 90%, a minimum acceptable might be 60%)
# What is the System Usability Scale (John Brooke's SUS)
# It the Trustmark discoverable and self-describing. (90%, 70%)
# Does the user feel safer as a result of the appearance of the Trustmark. (99%, 80% of those answering in the affirmative above.)
# Does the user feel that the site is safe overall? (the metric is a comparisons of the positives to the negatives.)
# Does the user understand the necessity for a strong identity for their providers?
# Does the user know whether the identity of the provider is strongly bound to a real-world entity?
# Collected verbatim responses used for site improvement.

===The System Usability Scale===
The SUS is a 10 item questionnaire with 5 response options.
#I think that I would like to use this system frequently.
#I found the system unnecessarily complex.
#I thought the system was easy to use.
#I think that I would need the support of a technical person to be able to use this system.
#I found the various functions in this system were well integrated.
#I thought there was too much inconsistency in this system.
#I would imagine that most people would learn to use this system very quickly.
#I found the system very cumbersome to use.
#I felt very confident using the system.
#I needed to learn a lot of things before I could get going with this system.
[[Category:User Experience]]
[[Category:Concept]]