NIH Generic OMB Clearance for Online Surveys

FAQ - Background on the OMB Clearance and its Relationship to Usability Testing

Please direct additional questions to Mangala Manivannan.

• Need for clearance: Why do we need an OMB clearance to put an online survey on our Web site?



• Statistical analysis: What kind of statistical analysis can we perform on the survey responses?



• Timetable: How long will it take for us to receive clearance on our survey instrument?



• Focus groups: Does this generic clearance address focus group evaluations or usability task analyses?



• Site content: Can we use this clearance to measure user satisfaction with the content of our Web site as well as the navigation?



• Program data: Can we gather new program data under this clearance?



• New Web pages: Can we use this clearance to help us design a new Web page, application, product, or service offering?



• Off-line surveys: Can we perform off-line surveys, distributed in hard-copy, under this clearance?



• Response time: How can we ask all the questions we need to ask and still stay within the 5-minute limit on response time?



• Demographic data: Using this clearance, can we find out who is visiting our Web site?



• Cookies: Can we use cookies as part of our survey?



• E-mail addresses: Can we collect e-mail addresses under this clearance?



• Social security numbers: Can we collect social security numbers and other sensitive information under this clearance?



• Race and ethnicity: Can we ask respondents to report their race and ethnicity under this clearance?



• Number of respondents: How many responses can I reasonably expect from an online survey?



• Paying respondents: Can we offer respondents a payment or gift as an incentive to complete the survey?

Why do we need an OMB clearance to evaluate our Web site?

OMB regulations prevent us from asking the same questions of more than 9 members of the public (non-Federal employees) unless we have the appropriate clearance to do so.

OMB encourages Federal agencies to use automated information technology to "reduce public burden" when gathering response data. If your online survey or Web poll can be accessed by the public, it must bear the OMB clearance number and expiration date signifying that the survey form has been reviewed by OMB and that it does not place "undue burden" on the public.

If you think that a pool of respondents recruited from Federal employees could provide you with valuable feedback on your Web site, and you devise an online survey that is functionally unavailable to the general public, then OMB clearance is not required.

What kind of statistical analysis can we perform on the survey responses?

We are not cleared to employ formal statistical methods in the analysis of survey data. You can use descriptive and summary statistics to evaluate results.

How long will it take for us to receive clearance on our survey instrument?

It takes 4 to 6 weeks to receive clearance from the time you submit your survey instrument and form to OLIB.

We submit your request package via e-mail to NIH/OD/OER/OPERA, they review it, print out three copies, and send it via courier to DHHS. From there, the package is sent to OMB. This process takes about two weeks. Once OMB has your submission in hand, they are obligated to review your request and provide a response within ten days. You will receive an e-mail from OLIB once your survey has been cleared.

Does this generic clearance address focus group evaluations or usability task analyses?

No, this clearance pertains only to online surveys. However, it is useful to know that OMB clearance for focus groups and usability studies is not required as long as you do not have more than nine participants and do not repeat the identical questions or script with other groups of nine.

Can we use this clearance to evaluate user satisfaction with the content of our Web site as well as the navigation?

Yes, with caution. NIH's request for a generic clearance was written in broad language to ensure that all aspects of a Web site could be evaluated using this mechanism:

The purpose of the proposed research is to ensure that intended audiences find the information provided on the Internet sites easy to access, clear, informative, and useful. Specifically, the research will examine whether the information is presented in an appropriate technological format and whether it meets the needs of users of these Internet sites. The research will also provide a means by which to classify visitors to the NIH Internet sites, to better understand how to serve them.

You can evaluate both content and navigation of your Web site under this generic clearance, however, you must be careful in how you word your survey questions. OLIB has created a repository of cleared surveys for your reference, and we are happy to work with you to develop OMB-compliant survey questions that will capture the data you need.

Note that this clearance cannot be used for off-line surveys, or Web-based surveys of off-line content. For example, under this clearance you can use an online survey to help evaluate a Flash multimedia product that is presented on your Web site, but you cannot use an online survey to evaluate user satisfaction with the same product distributed on CDROM or DVD.

Can we gather new program data under this clearance?

No. The generic OMB clearance was designed to help you evaluate your Web site as a communications tool, measuring you constituents' satisfaction with site navigation and content in terms of quality, clarity, and relevance. The clearance is not a license to gather new program data simply because it's done via the Web.

For example, if your Web site provides information on the health risks associated with poor eating habits, and you want to determine whether it's conveying the information effectively, you can evaluate this under the generic OMB clearance. You may not, in this example, use the clearance to survey your site visitors about their eating habits and disease conditions in order to come to scientific conclusions about the sample population. A study of this nature would require a separate clearance request.

Can we use this clearance to help us design a new Web page, application, product, or service offering?

Yes, with caution. The official title of NIH's generic OMB clearance is "Evaluation of User Satisfaction with NIH Internet Sites," which offers an important clue to understanding the kind of questions that you can ask. A survey of this kind measures site visitors' attitudes toward existing products, tools, and services on existing Web pages.

You may not ask users how they would feel "if" you made changes, for example, "Would this page be more user-friendly if there were more graphics on it?" or "Would you prefer to view this multimedia presentation in a version compatible with RealPlayer?" This crosses the line into "market research," and the Federal government has strict policies against conducting marketing-related activities. However, you may build a prototype Web offering (for the examples above, a live graphic-rich page or a RealPlayer version) and then ask users to respond to a survey rating their satisfaction with the proof of concept.

Certain kinds of planning information can be gathered without building a prototype, using carefully constructed questions in a "Request for Comment." This mechanism does not require OMB Clearance. Contact Mangala Manivannan for more information on this approach.

Can we perform off-line surveys, distributed in hard-copy, under this clearance?

No. Our approved clearance request states that "survey instruments will be conducted over the Internet" in order to "reduce the burden on the public."

If you wish to use print media in conjunction with your online survey, we suggest that you use it to alert potential respondents to the existence of your online survey by offering a URL that points them to the appropriate Web page.

How can we ask all the questions we need to ask and still stay within the 5-minute limit on response time?

• Employ multiple choice questions using radio buttons:
  
  Sample Question: Which funding mechanism supports your research?

NIH Grant NIH Fellowship NIH Career Development Award NIH Contract none of the above

In this example, the default choice is that no option has been selected.  If the user does not choose to answer the question, and does not click on an option, the database will record that no selection was made.  If the user clicks on a selection unintentionally, they cannot "unclick" that selection. The database will record the response next to the radio button they clicked on. That's why it's important to include a "none of the above" option when you employ radio buttons.

• Design your radio button interface in a matrix, so that users don't waste time scrolling:
  
  Sample Question: How well did each of the following multimedia site functions operate?

		Did not work	|	Not very well	|	Fairly well	|	Very Well	|	Did not use
	Audio
	Viewing slides
	Downloading copies of slides
	Viewing transcripts
	Searching transcripts
	Listening to transcripts on screen-reader

• Use "branching" or "skip logic" when programming your survey:
  
  Skip logic enables you to offer respondents one question at a time. The choice they make guides them to a series of questions related to that particular response, allowing them to "skip" over questions that don't pertain to their experience.
  
  Sample Question: Which version of the Web page did you view?

	Text-only	< Leads to questions about usefulness of content
	Multimedia	< Leads to questions about graphics and usefulness of content

We recommend that you test your survey functionality with a few users to see how long it takes them to complete it. As long as the average response time is 5 minutes, as stated in the burden disclosure language on your survey, you're within OMB guidelines.

For an in-depth look at designing user-friendly Web forms, consult the Usability.gov guidelines at http://usability.gov/pdfs/chapter13.pdf.

Can we find out who is visiting our Web site under this clearance?

Yes. You may use a reasonable number of demographic questions to get an idea of who is visiting your Web site. There is no actual limit on the number of questions you can ask, but each must be shown to contribute to an evaluation of Web site content or navigation, and not to provide new program data. You will be called upon to justify your use of demographic questions when submitting your survey form for OMB review.

Can we use cookies as part of our survey?

No. You may not use cookies to record information about respondents - with or without their knowledge. If your Web site relies on cookies for its function, we ask that you provide a link on your survey form to a privacy statement that informs the public about the use of cookies on your site. You may not use any information resulting from your Web site's cookies to correlate to survey response data.

NOTE: You may be aware that in FY2003, an NIH Institute applied to DHHS for permission to employ cookies so that site visitors weren't presented with a pop-up more than once. Please note that while this Institute did receive permission to use the cookie, two subsequent requests to use cookies have not been approved by DHHS.

Can we collect e-mail addresses under this clearance?

No. You may not specifically ask for respondent e-mail addresses. If, in the process of filling out your survey, a user volunteers an e-mail address and asks you to respond to a request for information, you may store the e-mail address for as long as it takes you to respond. This approach is part of our "assurance of confidentiality," as set forth in our clearance justification:

Demographic information will be used for analytical purposes only. Names will not be recorded on the questionnaires, nor will personal identifying data be maintained in the database. While respondents may be given the option of providing their e-mail addresses for the purpose of receiving a response to a particular inquiry, these addresses will not be used for purposes of linking particular respondents to particular survey responses. The addresses will not be retained after the reply is sent to the respondent.

Can we collect social security numbers and other sensitive information under this clearance?

No. We are not allowed to ask for "sensitive information" under this clearance. Sensitive information includes respondents' social security numbers, household income, and other information that is usually considered "personal."

Can we ask respondents to report their race and ethnicity under this clearance?

Yes, with justification. You can ask for respondents' race and ethnicity if you need it in order to understand response data. You will be asked to justify the need for this data, along with any other demographic data you ask for, on the submission form.

You must structure your question in the following way:

Race and Ethnicity I do not wish to provide this information			<	You must provide this option.
	Ethnicity: Hispanic or Latino Not Hispanic or Latino		<	Ethnicity designation must be asked first if provided as a separate set of choices. Ask your programmer to default both radio buttons to "unchecked."
	Race: Check all that apply. Click here for definitions. American Indian or Alaska Native Asian Black or African American Native Hawaiian or Other Pacific Islander White		<	Provide a link to an HTML page or pop-up that offers definitions of each race. Definitions can be found in OMB Directive 15.

How many responses can I reasonably expect from an online survey?

The number of responses you'll receive depends on how you publicize the survey. The Submission form offers you multiple notification methods to choose from. Choose as many as you think will offer you the number of responses you need.

• All site visitors will access the survey from a graphic or textual link on the Web page.
  
  Over the last two years of posting surveys online, NIH ICs have achieved very few results with this notification method.
  
  If you choose to use it, make sure that the link is placed high on the page. Use some kind of graphic treatment to make it stand out. And, try placing the link on multiple pages deep within your content, so that site visitors are presented with the opportunity to respond after they've navigated your Web pages. For example, if you want to survey users of a database, put a link or the search results page.



• All site visitors will be presented with a pop-up window that allows them to link to the survey.
  
  Pop-ups generate a much higher response rate for line surveys because they grab site visitors' attention. Use of a session cookie ensures that a visitor sees the pop-up only once. However, if a visitor checks back in with you site later on, they will see the pop-up again.
  
  Please consider using a small pop-up to provide a link to the survey, rather than launching a large pop-up containing you actual survey form. The smaller pop-up may prove less aggravating to site visitors and it won't obscure as much of the page.



• Random site visitors will be presented with a pop-up window that allows them to link to the survey.
  
  Randomizing the pop-up reduces the risk that a visitor will see it over the course of several visits to your site.
  
  One creative approach is to launch the randomized pop-up with a window that reads, "If you'd like to provide feedback on our Web site at the end of your session, please enter your e-mail address." The user can either dismiss the window by clicking a "No, thanks" button, or enter their e-mail address and click "Continue." They continue browsing your site, while your Outlook inbox receives the e-mail and sends an auto-reply containing a link to the Web page with the survey on it. This has the advantage of allowing site visitors to perform their tasks in peace, and respond to the survey after having navigated your site.
  
  Should you choose this method, please have OLIB review the language you use on your pop-up window and in your auto-reply so that your process complies with the Privacy Act. Contact Mangala Manivannan for guidance.



• An e-mail message will be sent to pre-existing distribution lists offering potential respondents a direct link to the survey.
  
  This is an excellent way to get results if you have a loyal audience already familiar with your site. Please be sure to uphold the Privacy Act if you choose this notification method.
  
  In order to comply with the Privacy Act, site visitors must be told what they're signing up for when they give you an e-mail address. The next time you send out your e-newsletter, add a line at the bottom that asks, "If you would like to participate in an online survey to help us improve our <Web-related activity>, please send us your e-mail address." That way, you can build a Privacy Act-compliant distribution list of people who are likely to provide feedback.



• The survey will not be publicly accessible, but will be available only to constituents contacted via e-mail.
  
  Use this notification when you want responses about a specific Web-related activity from a specific audience. for example, if you offered principal investigators the opportunity to sign up online for a mentoring program, and you wanted feedback about how easy it was to use the online form, you would not want the world at large to respond to an online survey. Bury the survey form within the Web site and e-mail constituents with a link to it.

As part of this clearance activity, the Online Information Branch is required to track the "total burden placed on respondents." The target number of respondents you supply is used to calculate this burden figure. We add the burden for your survey to the burdens of all other NIH online surveys under the clearance, and report the total to OMB upon request.

It is important that you track the number of responses your survey receives once it is launched from your Web site, so that you do not exceed your target number. Incomplete responses count.

If you decide to remove your survey before you reach your target, please send e-mail to Mangala Manivannan with the actual number of responses recorded.

Can I offer respondents a payment or gift as an incentive to complete the survey?

No. We do not have clearance to offer any type of remuneration to respondents.