Pros, Cons, and Best Practices of Job Simulations in Federal Hiring, Schemes and Mind Maps of Technology

Download Pros, Cons, and Best Practices of Job Simulations in Federal Hiring and more Schemes and Mind Maps Technology in PDF only on Docsity! JOB SIMULATIONS: TRYING OUT FOR A FEDERAL JOB A Report to the President and the Congress of the United States by the U.S. Merit Systems Protection Board A Report to the President and the Congress of the United States by the U.S. Merit Systems Protection Board JOB SIMULATIONS: TRYING OUT FOR A FEDERAL JOB U.S. Merit Systems Protection Board Neil A. G. McPhie, Chairman Mary M. Rose, Vice Chairman Office of Policy and Evaluation Director John Crum, Ph.D. Project Manager Laura Shugrue Executive Summary A Report by the U.S. Merit Systems Protection Board i The U.S. Merit Systems Protection Board (MSPB) has published a number of reports that highlight the importance of using good assessment tools. We have examined several assessment tools to determine their ability to improve the quality of agencies’ employee selections, including structured interviews, reference checks, and the probationary period. Job simulations, may also be an effective way for agencies to identify high-quality applicants. The purpose of this report is to examine the utility of job simulation assessments for use in the Federal hiring process. A job simulation is an assessment that presents applicants with realistic, job-related situations and documents their behaviors or responses to help determine their qualifications for the job. Job simulations include, but are not limited to, work samples, situational judgment tests, assessment centers, and job tryout procedures. Executive Summary Findings Job simulations present a number of important advantages as well as several significant disadvantages. Advantages • Higher Validity. Job simulation assessments tend to have higher predictive validity than many other types of assessments, meaning they should be better at predicting future job performance. • Better Job Fit. By exposing applicants to the types of events, scenarios, and challenges confronted on the job, job simulations help applicants determine if the job is well suited to their knowledge, skills, abilities, and interests. • Positive Applicant Perceptions. Because job simulations replicate the types of tasks performed in the actual job, studies have found that applicants are more likely to view them as being fair and job-related. • Greater Degree of Fairness. Research generally supports the premise that job simulation assessments have lower rates of adverse impact (i.e., a different rate of employment selection that works to the disadvantage of members of a race, sex, or ethnic group), as well as a lower degree of exposure to discrimination lawsuits based on the selection procedure. Executive Summary ii Job Simulations: Trying Out For A Federal Job Disadvantages • Cost. Job simulations are sometimes more costly because of the resources necessary to develop and administer the tests. The more advanced simulations require greater expertise to develop than other, less complicated assessments, and this expertise results in higher developmental costs. Plus, these advanced assessments can require more staff and training to administer and assess the results. • Limited Scope. While job simulations can be used to assess multiple competencies, a single simulation exercise will often focus on a limited number of tasks or duties performed on the job. Therefore, job simulations are best used as one assessment in a series of valid selection tools. • Not Suited to All Jobs. Many job simulations require the applicant to already have a certain level of knowledge, skills, or abilities (KSAs) to complete the assessment. Therefore, they may not be appropriate for some entry-level or generalist occupations. Agency Use of Job Simulations Job simulations are not being used extensively in the Federal Government, with less than half of the organizations we questioned indicating that they use them. The top barriers cited were related to a lack of resources and knowledge rather than amenability. This finding suggests that agencies might increase the use of job simulations if they had more knowledge or training in developing and administering these assessments and more resources to devote to these steps. Organizations that use job simulations the most tend to use them for mission- critical occupations or specialty areas (such as leadership). They also tend to be fairly homogenous agencies that do volume hiring into one or two entry-level, mission- critical occupations. Finally, agencies that use them tend to have budgets committed to assessment activities, dedicated assessment staff, and/or leaders who support devoting resources to workforce management issues. In terms of who develops and administers job simulations, agency selecting officials and subject matter experts seem to take the lead. Unfortunately, evidence suggests that they may not receive the training necessary to take on these responsibilities. Most of the organizations that used job simulations in the past year believed that the simulation tests increased the quality of the agency’s selections. However, few organizations reported actually measuring the impact. Executive Summary A Report by the U.S. Merit Systems Protection Board 1 The Federal Government is currently experiencing a surge in interest from potential applicants. As the number of applicants increases, agencies need to be careful to use good assessment tools to distinguish the most qualified applicants. Job simulations may help accomplish this goal. While many once considered the Government as an employer of last choice, many applicants now view a Federal job as a good employment opportunity. Some view it as a chance to effect change, perhaps because of President Barack Obama’s encouragement to young people to get involved in Government at every level and his determination “to make Government cool again.”1 Others view Federal employment favorably because the economy has not been kind to many private and nonprofit sector employees, and they are looking for the type of job security and stability the Government can often provide. Whatever the reason, the Federal Government has seen an increase in applicant interest. For example, by December 2008, 330,000 people had applied for positions within the Obama Administration. That compares to 44,000 applicants in President George W. Bush’s 2000-01 transition and 135,000 in President Clinton’s 1992-93 transition.2 At the same time, the Office of Personnel Management (OPM) reported that the number of visitors per month to USAJOBS (www.usajobs.gov)— the Federal Government’s official jobs Web site— almost doubled from the previous year, to 13.6 million visits. Many agencies have noted increased interest as well. For example, the Government Accountability Office (GAO) saw a 150-percent increase in applications, including 1,300 applicants for positions related to studying the Federal bailout.3 Finally, a 2009 survey of university career center officials found that college students are becoming increasingly interested in Federal employment.4 1 Barack Obama’s comments at the “Service Nation Presidential Candidates Forum,” Columbia University, New York, NY, September 11, 2008. 2 Mimi Hall, “330,000 Applicants Vie for Administration Jobs,” USATODAY.com, December 12, 2008. 3 Stephen Losey, “HR Staffs Deluged by Millions More Applicants,” FederalTimes.com, December 21, 2008. 4 Steve Vogel, “Survey Finds College Students More Interested in Federal Jobs,” Washington Post, May 1, 2009. Introduction Background 2 Job Simulations: Trying Out For A Federal Job Introduction These facts present a snapshot in time. We do not know how job seekers will rate the Government in the future. A year or two from now, Federal recruiters may find themselves standing alone at job fairs while applicants once again swarm Wall Street or the next cutting edge employer that offers huge salaries, signing bonuses, and prestige. What we do know, however, is that the Government always needs qualified people who have the ability and motivation to carry out its many missions. For this reason, the Government needs to ensure it is hiring the right people, and that is where good assessment practices come into play. The merit system principles (MSPs), found in 5 U.S.C. § 2301, constitute the framework for Federal human resources (HR) management. The second MSP states that “selection and advancement [for Federal jobs] should be determined solely on the basis of relative ability, knowledge, and skills.”5 Assessment is the phase in the hiring process when agencies determine the extent to which applicants possess the knowledge, skills, and abilities required to do the job. The purpose of assessment is to identify the best-qualified applicants. The MSPB is responsible for conducting special studies of Federal HR practices, policies, and procedures to ensure that they adhere to the merit system principles and do not result in prohibited personnel practices. As part of this mission, we have published a number of reports that highlight the importance of using good assessment tools. Specifically, we have looked at several assessment tools to determine whether they improve the quality of agencies’ employee selections, including structured interviews, reference checks, and the probationary period.6 The purpose of this report is to examine the utility of job simulation assessments for use in the Federal hiring process. For the purpose of this report, a job simulation means an assessment that presents applicants with realistic, job-related situations and documents their behaviors or responses to help determine their qualifications for the job. Job simulations include but are not limited to work samples, situational judgment tests, assessment centers, and job tryout procedures. We focus specifically on how agencies use these assessments for hiring new employees rather than for promoting employees. This report: 5 5 U.S.C. § 2301(b)(1). 6 Go to www.mspb.gov and select “MSPB Studies” to read MSPB studies and newsletters. Specifically, see the following reports we have published concerning assessment methods: The Federal Selection Interview: Unrealized Potential, February 2003; Reference Checking in Federal Hiring: Making the Call, September 2005; and The Probationary Period: A Critical Assessment Opportunity, August 2005. Purpose and Methodology A Report by the U.S. Merit Systems Protection Board 3 Introduction • Examines the advantages and disadvantages of using job simulations; • Reviews how Federal agencies develop and use job simulations to assess applicant qualifications; • Determines whether this type of assessment can help agencies improve the quality of their employee selections; and • Introduces readers to simulation tools that they may not be familiar with and directs them to resources with more information about how to develop and use them. This study relied primarily on the following sources of information: • Literature Review. We conducted a review of literature related to job simulation assessments, including work samples, situational judgment tests, assessment centers, and job tryout procedures. In addition, we examined professional journals and other articles related to overall trends and innovations in employee assessment. • Agency Questionnaires. We sent questionnaires to 28 Federal agencies asking them about their assessment practices and, in particular, how they use job simulation assessments.7 We asked agencies to ensure that individuals with knowledge of the agencies’ assessment practices complete the questionnaires. We received responses from 20 different Federal agencies, including 12 Cabinet level departments and 8 independent agencies. Because agency assessment programs are often delegated to the sub-component level (e.g., the Veterans Health Administration in the Department of Veterans Affairs or the Internal Revenue Service in the Department of the Treasury), we received multiple responses from some agencies. Therefore, we analyzed a total of 35 questionnaire responses, including responses from the department and/or sub-component level. • Interviews with Employers. To follow up on the results from the questionnaire responses, we interviewed four organizations that had particularly interesting examples of how they have used job simulation assessments. These examples are highlighted in Appendix A. • Interviews with OPM. We conducted interviews with colleagues in two OPM offices to gain more information on research and policy work that OPM has done regarding job simulations, as well as to find out what experiences they have had helping agencies implement these types of assessments. 7 See Appendix B for a copy of the interrogatory and the instructions. 6 Job Simulations: Trying Out For A Federal Job Principles Of Assessment Research also indicates that selecting the wrong applicants can increase costs and decrease productivity. Some estimates show that the cost of hiring a person who does not have the right KSAs is up to three times the employee’s salary in terms of wasted hiring time, training costs, salary, benefits, and severance pay.14 Poor selections can also result in lost productivity because of the time it takes to correct performance and conduct problems. For instance, a 2004 study found that managers in the United States spend about 13 percent of their time correcting employees’ mistakes.15 Other research has found that poor selection can lead to increased turnover and absenteeism.16 To reduce the costs associated with bad selections and improve the organization’s ability to carry out its mission, it is important to employ good assessment strategies that help identify the best candidates for the job. Having reviewed the professional literature, there are several factors that appear to serve as the foundation of good employee assessment: The assessment should be reliable. Reliability means that the score an applicant receives on the assessment is consistent. For instance, if the applicant were to take the test more than once, with no performance intervention, the score should be similar each time. In short, reliability ensures that random sources of error are minimized during the assessment so that there is consistency and repeatability. The assessment should be valid. Validity is critical to determining which assessments will produce the greatest results. Validity is the relationship between performance on an assessment and performance on the job. Specifically, validity demonstrates if the assessment measures a job-related characteristic and how well it measures that characteristic. There are three key methods used to determine the validity of an assessment: • Content-related validation is evidence that the questions or tasks included in an assessment actually represent the knowledge, skills, or abilities necessary to perform the job. Content validity is generally based on the results of a job analysis and subject matter expert judgment. Content validity is the easiest to achieve and is, therefore, a popular method used in Federal assessment practices. 14 Corporate Leadership Council, Literature Review, “Employee Selection Tests,” Catalog No. 070-198- 213, Washington, DC, March 1998, p. 2. 15 The Future Foundation and SHL, Getting the Edge in the New People Economy, 2004, p. 31. 16 The Partnership for Public Service, Asking the Wrong Questions: A Look at How the Federal Government Assesses and Selects Its Workforce, Washington, DC, October 2004, p. 3. What Makes a Good Assessment? A Report by the U.S. Merit Systems Protection Board 7 Principles Of Assessment • Construct-related validation requires a demonstration that the assessment being used measures the construct (i.e., the psychological or physical characteristics that distinguish between people and how they will perform on the job, such as intelligence or sociability) it was intended to measure. • Criterion-related validation is the proven statistical relationship between performance on the assessment and performance on the job. It requires a validation study in which job incumbents are tested and the tests are compared to performance evaluation results (concurrent validity) or job applicants are tested and tests are later compared to performance results on the job (predictive validity). Because of its ability to best predict an applicant’s future performance on the job, criterion-related validity should be what organizations strive to achieve. As previously indicated, studies have shown that an increase in predictive validity will increase the percentage of new hires who will perform satisfactorily on the job.17 However, a validation study also requires a certain level of expertise to achieve, making criterion-related validation more resource intensive. The most commonly used measure of predictive validity is the validity coefficient. This coefficient ranges in absolute value from 0 to 1.00. The closer the score is to 1, the stronger the relationship between the assessment tool results and job performance. A validity coefficient of .30 or higher is generally considered desirable for purposes of employee assessment. The assessment should be used for the appropriate target population. Different assessments are appropriate for different situations. For instance, a job knowledge test that is designed to test an applicant’s knowledge of a specific field should not be used for an entry-level position that does not require prior job knowledge or experience. Another example is the multiple hurdle approach that MSPB has recommended in earlier reports. As part of this process, agencies use fast, accessible, and low-cost instruments in the early part of the assessment process, thereby saving assessments that have a higher cost to administer for later in the process when more precision is needed to identify the best candidate.18 Applicants should have positive perceptions about the assessment process. An applicant’s perception that an assessment is fair and job-related promotes cooperation and acceptance of the assessment process.19 Specifically, research shows that applicant perceptions of the selection process are important to their overall perception of the 17 Taylor and Russell, “The Relationship of Validity Coefficients to the Practical Effectiveness of Tests in Selection: Discussion and Tables,” pp. 565-578. 18 U.S. Merit Systems Protection Board, Reforming Federal Hiring: Beyond Faster and Cheaper, July 2006, pp. 27-28 19 U.S. Office of Personnel Management, Assessment Decision Guide, http://apps.opm.gov/ADT. 8 Job Simulations: Trying Out For A Federal Job Principles Of Assessment organization and whether they want to work for it. Results from a meta-analysis of 86 independent studies demonstrate that applicants with favorable perceptions of the selection process were more likely to view the organization favorably. Therefore, they were more likely to accept job offers and to recommend the employer to others.20 We know from previous MSPB research that word-of-mouth is a key way applicants find out about Federal employment opportunities, so these recommendations are important. The assessment should be fair and unbiased. It is important to use assessment tools that are as unbiased and fair as possible and that do not result in adverse impact. Adverse impact means that the assessment creates a substantially different rate of selection for individuals of a particular ethnicity, race, or gender. To help hiring organizations comply with this principle and make proper use of tests and other selection procedures, the Equal Employment Opportunity Commission, the Department of Labor, the Department of Justice, and the former Civil Service Commission developed the Uniform Guidelines on Employee Selection Procedures.21 It is important to ensure that assessment practices conform to these guidelines, not only to help improve applicant perceptions of the assessment process, but also to help save the organization from unnecessary litigation over selection practices. In the next section, we discuss the different kinds of assessment methods that fall under our definition of “job simulation.” 20 John P. Hausknecht, David V. Day, and Scott C. Thomas, “Applicant Reactions to Selection Procedures,” Personnel Psychology, Autumn 2004, Vol. 57, No. 3, p. 639. 21 http://www.uniformguidelines.com/uniformguidelines.html. A Report by the U.S. Merit Systems Protection Board 11 Job Simulations: What They Are Positive Applicant Perceptions. Because job simulations replicate the types of tasks performed in the actual job, applicants are more likely to view them as being fair and job-related. As previously noted, positive perceptions of the assessment can improve the result of the hiring process. In addition, studies have shown that applicants who view the assessment as being relevant to the job may work harder to perform better on the assessment. Greater Degree of Fairness. Research generally supports the premise that job simulation assessments have lower rates of adverse impact, as well as a lower degree of exposure to discrimination lawsuits, based on the selection procedure. This not only saves the employer resources having to defend assessment tools in court, but it also reinforces applicants’ positive perceptions of the process. Overcome Faults of Other Assessments. Job simulations can help overcome the faults of non-simulated assessments, such as an applicant’s distortion of his or her own abilities or “faking” competence. Also, job simulations rely on actual behavior rather than the verbal descriptions offered during an interview, which makes them more accurate measures of actual ability.27 Disadvantages Cost. Job simulations can cost more. The more advanced simulations require added expertise to develop than other, less complicated assessments, and this expertise would result in higher developmental costs. Plus, they can require more staff and training to administer and assess the results. The use of job-related material or equipment may also add to the cost. Limited Scope. While job simulations can be used to assess multiple competencies, a single simulation exercise will often, out of design necessity, focus on a limited number of tasks or duties performed on the job. If a work sample is used for a clerical position to measure an applicant’s ability to type a memo, the assessment could miss other key aspects of the job, such as customer service skills. Therefore, job simulations are best used as one assessment in a multiple hurdle process, where a number of assessments are used in succession to measure different dimensions of the applicant’s ability. Not Suited to All Jobs. Many job simulations require the applicant to already have a certain level of knowledge, skills, or abilities to complete the assessment. Therefore, they may not be appropriate for some entry-level or generalist occupations that do not require pre-existing knowledge. 27 Robert D. Gatewood and Hubert S. Feild, Human Resource Selection, 5th edition, Harcourt College Publishers, 2001, pp. 633-634. 12 Job Simulations: Trying Out For A Federal Job Job Simulations: What They Are Limitations of the Research. Some of the research reviewed for this study pointed to the fact that there were still many areas that need further research in terms of the utility of different job simulation methods. For instance, while some of the most referenced studies on work samples cite high levels of validity and low levels of adverse impact, some recent studies have questioned these research findings. We hope that bringing more attention to job simulation assessments will encourage the pursuit of additional research in this area. Below, we provide basic information on typical job simulations that organizations may want to add to their assessment tool kit. Appendix A of this report contains a more in-depth look at each assessment. The discussions in this report are not meant to be exhaustive. Many books and research studies have looked at different aspects of work samples, situational judgment tests, assessment centers, and job tryouts. We are providing context and generalized research findings so that readers can become familiar with the advantages and disadvantages of these assessments and make educated decisions on what is appropriate to pursue for their organization. Types of Job Simulations WORk SAMPLE TEST Definition Examples Advantages Disadvantages Evaluates applicants’ job- related skills by having them perform actual activities or tasks that are physically or mentally similar to the duties they would perform on the job. A work sample assessment for a welder may include an onsite welding exercise. Most applicants will already have the basic knowledge to perform the task. Therefore, test developers just need to ensure the equipment is available and develop measurements to evaluate the products produced. A work sample for a customer service job might involve role- play between the applicant and an “angry customer” (the assessor). The applicant may sit at a desk with a telephone, computer, and access to necessary records and standard operating procedures to be able to respond to the scenario. The caller would call with a complaint and evaluate the applicant’s responses against pre-determined benchmarks. They are generally viewed as having a high validity rating (.54), meaning that they are good predictors of future job performance. They are generally viewed as having low adverse impact and stand up well in selection discrimination litigation. Applicants are less able to “fake” proficiency. Applicants view them as fair because they can see the relationship to the job. Work sample tests provide applicants with a job preview to better inform their decision on whether they are a good fit for the job. They generally do not measure aptitude or future potential. Their scope is limited to only the competencies needed for the specific activity carried out during the test. They are not very useful for tasks that take a long time to complete. A Report by the U.S. Merit Systems Protection Board 13 Job Simulations: What They Are SITuATIOnAL JuDGMEnT TEST (SJT) Definition Examples Advantages Disadvantages Presents applicants with a description of a work-related scenario and asks them to exercise their judgment by choosing, constructing, or evaluating alternative courses of action given the situation. An applicant may be provided a paper and pencil test that contains several detailed scenarios. After reading through the scenarios, the applicant is asked to respond to the scenario. The following is a brief example of a possible SJT question: You have noticed that there is a lack of cooperation and trust among the employees in the organization you manage. This counter-productive behavior is adversely affecting both the quality and the quantity of the work produced by the employees. You would most likely… a. Inform the employees that failure to cooperate and work harmoniously will result in disciplinary action. b. Bring in someone to do some team building with the employees. c. Meet with the employees to discuss your observations and identify the cause of the interpersonal problems. d. Change some of the work assignments in order to attain a more cohesive and cooperative work group.* They have moderately high levels of predictive validity (about .34). SJTs are easily administered and scored, especially when using video-based or digital technology to administer and record answers. Applicants can see the link between SJT scenarios and the work. As a result, they take the test seriously and try to do well. Also, the test provides a good preview of what the job will be like. SJTs are generally not susceptible to “practice effects” (i.e., improved performance on the assessment if the applicants complete it more than once). It is harder for dishonest applicants to remember and disclose to other applicants the longer SJT questions. SJTs have lower adverse impact than many written tests (e.g., cognitive ability tests), particularly when video or digital technology is used to administer the test. SJTs help measure applicants’ responses to ambiguous conditions. Good SJT scenarios contain a rich set of details, only some of which help determine the correct answer. If the test provides insufficient detail, the question does not fully test an applicant’s ability to discern the relevant information and respond accordingly. If correct answers are too obvious, the SJT can become a test of what applicants know they “should” do on the job rather than what they would actually do. For some scenarios, four or five answer options might not cover the full range of actions, not capturing what the applicant might really do— right or wrong— on the job. *Example taken from Dennis A. Joiner, “Assessment Center Trends,” Presentation at IPMA-HR Assessment Council Conference on Personnel Assessment, June 23, 2004. Executive Summary A Report by the U.S. Merit Systems Protection Board 17 Job Simulations: How Federal Agencies Use Them Federal agencies have a number of tools available to them to assist in evaluating candidate qualifications. Some are better than others in predicting applicants’ future performance on the job. To use these tools effectively, agencies need to have adequate resources, expertise, and an understanding of employee assessment principles and practices. To understand how agencies are using job simulation assessments, we queried them about their typical assessment practices. nature of Assessment Programs Figure 1. Agency Satisfaction with Applicants and New Hires 6% 56% 34% 3% 13% 66% 22% 0% 10% 20% 30% 40% 50% 60% 70% Excellent Very Good Good Fair Poor Applicants New Hires Satisfaction with Applicants and new Hires. In the questionnaire we sent to Federal agencies, we asked about the quality of the agency’s applicants and new hires. As seen in Figure 1, respondents were generally positive about the quality of the applicants for their organizations’ vacancies, with 62 percent rating them as very good to excellent. Similarly, respondents were positive about the quality of the new employees selected by their organization. Over three-fourths rated them as very good to excellent. The numbers for new employees are in line with past research the MSPB has done with • 18 Job Simulations: Trying Out For A Federal Job Job Simulations: How Federal Agencies Use Them supervisors having reported similarly high satisfaction with new hires.28 While the levels of satisfaction depicted in Figure 1 are impressive, there is still room for improvement. For instance, it would be in the Government’s interest if more agencies rated applicants and new hires in the “excellent” category. Furthermore, one-third of agencies rated applicants only as “good,” and almost a quarter rated new employees as only “good.”29 In this regard, MSPB research has found that Federal agencies often use selection methods that are not very capable of distinguishing between the good candidates and the superstars.30 It is possible that if agencies used better assessment methods, selecting officials would be even more impressed with applicants and new hires. Roles and Responsibilities. Ideally, HR staffs and selecting officials work closely together to identify, develop, and administer the tools used to assess applicant qualifications because HR staffs are often the assessment and selection experts in the agency and selecting officials are the subject matter experts (SMEs). We asked questionnaire respondents a number of questions related to the roles and responsibilities of HR staffs and selecting officials in the assessment process. First, we asked what level of knowledge HR staffs and selecting officials have about assessment tools. Keeping in mind that the questionnaires were generally completed by HR staff members, the results indicate that HR staffs are more knowledgeable about assessment tools than selecting officials. Eighty-four percent said HR was knowledgeable or very knowledgeable while 44 percent rated selecting officials the same. Second, we asked who is typically responsible for deciding what assessment tools will be used. This is an area in which we would like, and expect, to see shared responsibility among the different groups involved in the selection process, and our expectation was borne out by the data. Few agencies relied on only one group to carry out this responsibility. Generally, HR had the highest level of responsibility for deciding what assessment tools would be used to measure applicant qualifications (79 percent of respondents said HR is typically responsible for deciding what assessment tools will be used), but there was some sharing of this role with selecting officials (68 percent) and subject matter experts and/or line staff (52 percent).31 A few respondents noted that they have Industrial/Organizational (I/O) psychologists or other testing staff who participate (18 percent). Finally, almost one-quarter (24 percent) reported that OPM typically assists in this role. 28 For instance, see U.S. Merit Systems Protection Board, The Probationary Period: A Critical Assessment Opportunity, 2005; Building a High-Quality Workforce: The Federal Career Intern Program, 2005; Accomplishing Our Mission: Results of the Merit Principles Survey 2005, 2007; and In Search of Highly Skilled Professionals: A Study on the Hiring of Upper-Level Employees from Outside the Federal Government, 2007. 29 When we refer to “agencies” that responded to the interrogatory, this includes the total 35 responses received from the Department level and major sub-component level organizations. 30 MSPB, Reforming Federal Hiring: Beyond Faster and Cheaper. 31 Because respondents were instructed to select “all that apply,” percentages do not total 100. A Report by the U.S. Merit Systems Protection Board 21 Job Simulations: How Federal Agencies Use Them Table 1. Types of Job Simulations Agencies Used in Past Year Simulation # of Agencies Work Sample 9 Assessment Center 7 Situational Judgment Test 5 Job Tryouts 0 Most organizations that used job simulations used them for less than 10 percent of their vacancies in the past year, as shown in Table 2. These agencies generally have a variety of mission-critical occupations (only some of which are amenable to job simulation assessments) or are using simulations for specialty areas (such as leadership positions). The agencies that use them for more than 50 percent of their vacancies are fairly homogenous agencies that do volume hiring into one or two entry-level, mission-critical occupations, such as law enforcement or foreign service officers. Table 2. Percent of Vacancies for which Agencies Used Job Simulations in Past Year Percent of Vacancies number of Agencies Less than 10 10 11-25 2 26-50 1 51-75 1 More than 75 2 Deciding to use Job Simulations. We asked agencies how they determine whether to use simulations, and they cited a number of factors that influence this decision. First, agencies noted that the results of the job analysis helped determine whether the position would be amenable to a simulation. Once they identified the key tasks of the job and the competencies and skills needed to carry out those tasks, it became easier to see what types of assessments would best measure those competencies. Budget was another driving force behind the decision. Because simulations generally require more resources to develop and/or administer, agencies have to ensure they have the budget necessary to support the assessment. Another factor appears to be the relationship between selecting officials and HR staffs. When a consultative relationship exists, there is more communication and discussion of what would be the best assessments to use for the position/occupation. In addition, such a relationship seems to foster more openness to using non-traditional types of assessments. 22 Job Simulations: Trying Out For A Federal Job Job Simulations: How Federal Agencies Use Them Jobs and Skills for which Simulations Are used. We asked agencies to identify the occupations for which they used job simulations and found that agencies used them for a wide variety of occupations. Some agencies used them for support positions such as clerical, human resources, or information technology positions. Other agencies used them for mission-critical occupations, such as those in health care or law enforcement. Some reported using them for all positions above a certain grade level, and others use them only for leadership positions. The types of competencies for which agencies used simulations were a bit easier to categorize, but still were varied across organizations. Agencies used them to measure competencies such as technical skills, customer service skills, communication, problem solving, decision-making, leadership, interpersonal skills, achieving results, and flexibility. How Job Simulations Are used. Over half of the respondents who reported that their organizations had used simulations in the past year indicated that they used them as one hurdle in a multiple hurdle process, as opposed to a stand-alone assessment. A multiple hurdle approach uses a set of valid assessment procedures successively to manage the candidate pool and narrow the field of qualified candidates.34 Research has found that using good assessment procedures in succession can improve the ability of the assessments to predict job performance, adding to the quality and cost benefit of the process. The Federal assessment process can be broken into two distinct steps. First, applicants are assessed to ensure that they meet the minimum requirements to perform satisfactorily on the job— what the Government refers to as minimum qualifications. Second, for those who meet minimum qualifications, their relative qualifications are assessed to make distinctions among the qualified applicants. Different assessment tools can be used to carry out these steps. All of our respondents reported using simulations for assessing relative qualifications, not for making minimum qualification determinations. Using simulations in this way is understandable. Because they are generally more resource intensive to administer, job simulations can cost more per applicant to administer. Therefore, using them to distinguish among the smaller pool of the most qualified applicants is a more judicious use of resources. Implementing Job Simulations. There are some key steps to implementing job simulation assessments: (1) the test developer is selected; if he or she lacks expertise, arrangements should be made for an expert to assist; (2) the test is developed; (3) the test administrator is selected and trained; (4) the test is administered; and (5) the test results are assessed. We asked agencies how they carry out these steps. We found that selecting officials have a lot of responsibility, but not necessarily a lot of assistance. 34 For more information on the multiple hurdle approach, see U.S. Merit Systems Protection Board, “Fewer Hoops, Higher Hurdles,” Issues of Merit, September 2002; “How Many is Multiple,” Issues of Merit, July 2003; and Identifying Talent through Technology, 2004, pp. 75-77. A Report by the U.S. Merit Systems Protection Board 23 Job Simulations: How Federal Agencies Use Them Almost half of the organizations that use job simulations indicated that the selecting official is typically responsible for developing the test. In these cases, the selecting official received no assistance from the HR staff or I/O psychologists, though some did receive assistance from a contractor or subject matter expert. This occurred despite the fact that few of the organizations that used simulations indicated that selecting officials are knowledgeable or very knowledgeable about the assessment tools available for measuring applicant qualifications. It is of concern that assessments are being developed and administered by individuals who may not have the expertise necessary to develop, administer, assess, and defend the tests. However, we must keep in mind that this was reported by only a few organizations and cannot be generalized to all Federal agencies and therefore do not provide enough evidence of a systemic problem. Nonetheless, we note that if selecting officials have significant responsibility for developing assessments, it is especially important for them to have assistance from assessment experts. Without the proper expertise, simulations could lead to erroneous or less than optimal selections and/or adverse impact. In terms of administering simulations, almost half of the organizations that used job simulations reported that the selecting official was typically responsible for administering the test. SMEs were involved in administering the simulations in about a third of the organizations. The fact that selecting officials and SMEs have a lot of responsibility in administering assessments is not in and of itself a problem, as long as those participants receive the proper training to carry out their responsibilities. Simulation assessments can require significant training to ensure that administrators understand their role in the assessment process. However, as Table 3 demonstrates, some organizations may not be providing sufficient training in the different phases of the assessment process. For instance, the two organizations that reported that the selecting officials administer the simulations with no assistance also reported that administrators receive no training. Table 3. Training Provided for Administering Simulations number of Days number of Agencies 5 or more days 2 2-4 days 2 1 day or less 3 No training 2 Don’t know 6 Results of Job Simulations. Most of the organizations that used job simulations in the past year believed that the simulation tests increased the quality of the agency’s selections. Unfortunately, few organizations reported actually measuring the impact. The organizations that did measure the effectiveness of these assessments generally 26 Job Simulations: Trying Out For A Federal Job Formulating An Assessment Strategy OPM defines job analysis as a “systematic procedure for gathering, documenting, and analyzing information about the content, context, and requirements of the job.”35 The Code of Federal Regulations requires that “[e]ach employment practice of the Federal Government generally, and of individual agencies, shall be based on a job analysis...”36 The Uniform Employee Selection Guidelines call job analysis essential to the professional practice of employee selection.37 The typical job analysis will determine the relative importance of the job’s duties and tasks and identify the KSAs necessary to perform them. In addition, the job analysis will identify other factors that impact the ability to carry out the job’s duties, such as tools and technology, work environment and relationships, and training and licensure requirements.38 Ultimately, the job analysis demonstrates the relationship between the duties and tasks performed on the job, and the competencies and KSAs required to perform the tasks. This information can then be used to inform a number of workforce management decisions, including the development of the position description, compensation, recruitment strategies, onboarding strategies, training and development needs, and deciding on the assessment methods to use. For the purpose of this report, the job analysis should be used to identify job content, the competencies required, and the assessment methods to use, as well as establish minimum standards for assessor training. A job analysis is carried out by a “job analyst.” This person must know the job well and/ or have extensive training in carrying out the analysis. If the person has only one of these two attributes, a team approach may be used. In fact, the most effective job analyses are often performed through cooperative ventures in which a subject matter expert provides the job knowledge and the HR staff or I/O psychologist translates that information into usable formats and applications. Many methods can be used to conduct a job analysis. These include direct observation of employees, incumbent and/or supervisor interviews, structured questionnaires, expert panels, task inventories, checklists, and work logs. The critical incident technique lends 35 Office of Personnel Management, Delegated Examining Unit Operations Handbook, 2007. 36 5 C.F.R § 300.103 37 Uniform Guidelines on Employee Selection Procedures, http://www.uniformguidelines.com/ uniformguidelines.html#88. 38 OPM’s Delegated Examining Operations Handbook 2007 provides detailed guidance and tools on how to conduct an appropriate job analysis. The handbook can be found at http://www.opm.gov/deu/Handbook_2007/ DEO_Handbook.pdf. (1) Conduct a Job Analysis A Report by the U.S. Merit Systems Protection Board 27 Formulating An Assessment Strategy itself to developing scenarios for many simulation exercises.39 A typical method of job analysis would be to collect information from a small group of job incumbents through interviews or focus groups. The resulting duties, tasks, and competencies are then distributed to a larger group of incumbents through a questionnaire. Those incumbents rate the importance and frequency of each duty and task and the importance and need of the related KSAs. The job analyst would then use the information to prepare a job description. While the method may differ, the job analysis should ultimately collect information regarding:40 • The basic duties and tasks of the position, including frequency, importance, duration, effort, skill, complexity, equipment, and standards. • The competencies and KSAs required to perform the job. Job analyses typically state the minimal requirements for the job, though some experts recommend identifying the requirements necessary to perform that job at a higher level and focusing on outcomes rather than duties and tasks. In particular, the critical incident technique is a process that identifies behavior or events that make a significant contribution to an activity, such as the KSAs that distinguish satisfactory performance from unsatisfactory performance. • The environment in which the duties are performed, particularly any physical requirements that may pose obstacles to accomplishing the tasks, such as any unpleasant or risky conditions. • The tools and equipment necessary to carry out the duties and a description of how they are used and what skill level they require. • The relationships required to carry out the job, such as supervision that is given or received or relationships with internal or external stakeholders. The results of the job analysis will identify the critical tasks of the job and the skills necessary to carry out those tasks. Once these have been established, the organization can use that information to identify which assessments would best measure the identified KSAs, including job simulations. For instance, if a job requires already established knowledge of particular tools or equipment, a work sample could help identify those candidates who already possess that knowledge. In another example, the job analysis may identify the critical incidents incumbents may find themselves in that require a high level of judgment. Those situations could be used as the basis of a situational judgment test. 39 For more information on this technique, see John C. Flanagan, “The Critical Incident Technique,” Psychological Bulletin, Vol. 51, No. 4, July 1954. 40 HR-guide.com, “Job Analysis: Overview,” Job Analysis Internet Guide, 1999, http://www.hr-guide.com/ data/G000.htm. 28 Job Simulations: Trying Out For A Federal Job Formulating An Assessment Strategy The results of the questionnaires and agency interviews we conducted demonstrate that the agencies that effectively use job simulations base them on up-to-date job analyses. Some of them update the analyses on a scheduled basis, some as necessary, but all indicated that the job analysis is an important step in identifying how to best use the simulations. Anecdotal observations from some of the agencies we interviewed, including OPM, indicated that the lack of an up-to-date job analysis is a key barrier to agencies’ ability to effectively establish a new assessment process, such as simulation assessments. A job analysis can take a significant amount of resources to complete; however, it provides the roadmap for the assessment process. Those involved in assessment and selection need to understand the job as it currently stands before they can determine how to fill it. By using an old job analysis, an organization may miss some key aspects of the job that it needs to consider when hiring a new person. Technology is redefining the work we do and how we perform it. Manual work is declining while knowledge-based work is increasing. These trends could easily impact the competencies needed to effectively carry out the duties of a position. Therefore, using an out-dated job analysis could result in hiring the wrong person with the wrong skills. Ensuring that job analyses are current can reduce the resources it will take later to try to train that person for the job or to have to re-recruit for the position. Following the job analysis, the agency will know what competencies and KSAs should be tested. The next step is to identify which assessments to use. As discussed earlier, one of the key barriers to using job simulations was lack of knowledge about them. Therefore, this is where organizations need to do their homework and educate the HR staff, selecting officials, and other agency leaders about assessments, including job simulations. The number of assessment tools available to organizations is extensive, and they all have their advantages and disadvantages. They differ in a number of areas, such as: 41 • Purpose— job selection, promotion, development; • What they are designed to measure— competencies, work styles, values; • What they are designed to predict— job performance, managerial potential; • Format— paper and pencil, video or online, role-play; • Assessment characteristics— level of standardization, objectivity, and quantifiability; and • Type of administration— individual versus group. 41 U.S. Department of Labor, Employment and Training Administration, Testing and Assessment: An Employer’s Guide to Good Practices, http://www.onetcenter.org/dl_files/empTestAsse.pdf. (2) Identify the Assessments A Report by the U.S. Merit Systems Protection Board 31 Formulating An Assessment Strategy Research in Economic Value-Added also suggests that employing a superior, rather than average, performer will result in huge economic and productivity gains.48 Furthermore, Schmidt and Hunter’s analysis of selection utility literature found that improving hiring methods typically leads to large gains in economic value and the gains are directly proportional to the size of the increase in predictive validity.49 They went so far as to say that “in economic terms, the gains from increasing the validity of hiring methods can amount over time to literally millions of dollars. However…by using selection methods with low validity, an organization can lose millions of dollars in reduced production.”50 To reduce the costs associated with bad selections and improve the organization’s ability to carry out its mission, it is important to employ good assessment strategies that help identify the best candidates for the job. While research indicates that using good assessment practices may increase the bottom line, the agency’s leadership ultimately decides how important assessment tools are compared to other priorities. To help ensure that agency leaders understand the impact of good assessment practices on agency results, HR staffs need to build the business case for making assessment a priority and investing the necessary resources to it, even when the best choices may be the more costly ones. Required skills. Some assessments are better than others for testing specific skills. For instance, because work samples often require that applicants already possess some of the knowledge, skills, or abilities necessary for the job, they may not be appropriate for measuring potential but may be good for a later phase of testing. A situational judgment test is an effective measure of social skills but is less likely to be effective at evaluating technical or motor skills. An assessment center can be used to test multiple competencies, but the resource requirements make them a better choice for mission- critical and/or high-volume occupations rather than for single vacancies. So, when choosing assessment methods, the organization must identify what it wants to measure and determine which assessment best evaluates that competency, knowledge, skill, or ability. Some of the time, a job simulation will be appropriate; other times, a non- simulation test will be best. Assessment quality. As discussed earlier in this report, there are several factors that serve as the foundation for high-quality assessments. The assessment should be: • Reliable— produce consistent results; • Valid— measure what it was intended to measure; and • Fair— be appropriate for the targeted applicant pool, be viewed by the applicant as unbiased, and reduce adverse impact. 48 Spencer, “The Economic Value of Emotional Intelligence Competencies and EIC-Based HR Programs,” 2001. 49 Schmidt and Hunter, “The Validity and Utility of Selection Methods in Personnel Psychology,” p. 263. 50 Schmidt and Hunter, “The Validity and Utility of Selection Methods in Personnel Psychology,” p. 273. 32 Job Simulations: Trying Out For A Federal Job Formulating An Assessment Strategy When developed and administered properly by qualified testing professionals, job simulations can meet all of these criteria. Number and placement of assessments. There are no rules about how many assessments an organization should use in the hiring process. It depends on the job, the competencies necessary to carry out the job, and the resources available. MSPB’s reports have long advocated a multiple hurdle approach to assessment. This type of approach can help avoid poor selection by using a set of relatively valid assessment procedures successively to manage the candidate pool and narrow the field of qualified candidates. Research has demonstrated that using good assessment procedures in succession can improve the ability of the assessments to predict job performance, adding to the quality and cost effectiveness of the process.51 Each assessment tool should be used to complement, not replicate, one another. They should be implemented in a timely manner so that good applicants do not drop out of the process. Ideally, assessment procedures should be selected and sequenced based on cost and benefit. For instance, tools that are less costly to administer should be used in the beginning stage of the assessment process when the candidate pool is largest. A situational judgment test could serve well as an early hurdle because it is effective at narrowing the candidate field while requiring fewer resources to administer and score. Performance-based simulations, such as assessment centers or work samples, might be more effectively used later in the assessment process because they can take more resources to administer to large groups of applicants though they make better distinctions among applicant qualifications. This type of multiple hurdle approach can help agencies effectively manage their resources while increasing the quality of their selection process. Number of expected applicants. To get the best return on investment for the assessment, the organization should consider how many people it expects will apply for the position. As mentioned above, we recommend selecting and sequencing assessment procedures based on cost and benefit. If the organization anticipates a large applicant pool, then a low-cost screening assessment would be appropriate to narrow the applicant field. However, if it is a hard-to-fill position that generally receives few applicants, then it makes sense to go straight to the higher cost but more precise assessment tool— saving time, money, and applicant burden by using fewer assessments. Applicant burden. Two of the key complaints about the Federal hiring process are its complexity and its length. Assessment procedures can add to both of these issues. Therefore, it is important to ensure that the assessment methods the organization chooses are implemented in a thorough but timely manner. In terms of the multiple hurdle process, while successive hurdles can improve the quality of the hiring process, excessive hurdles can impede the effectiveness of the hiring process. Too many hurdles can result 51 Schmidt and Hunter, “The Validity and Utility of Selection Methods in Personnel Psychology,” p. 265. A Report by the U.S. Merit Systems Protection Board 33 Formulating An Assessment Strategy in applicant attrition and do little to improve hiring decisions. An excessive number of assessments can also deter qualified applicants from applying because of the applicant burden. Applicant burden is an issue to consider when determining whether job simulations are appropriate. Job simulations, as well as other assessment tools such as in-person interviews, can present a significant burden to applicants if, for example, they require physical attendance at the test administration site. Applicants may then have to use their time and money to get to the location. Having a number of sites available or paying for applicant travel could help decrease the applicant burden but it would increase the agency’s administration costs. There are tools available to help organizations reduce applicant burden, as well as agency cost, when using job simulations. Two such tools are described below: Video/Digital Technology. The use of video or digital technology has been increasing over the past decade. Video or digital technology can be used in test administration to present the applicant with the test instructions, contextual information, practice scenarios, and/or the actual test scenarios, questions, or items. Video and digital technology can also be used to capture the applicants’ answers. Assessors can then evaluate candidates’ responses at a later, more convenient time and location or computers can be used to score multiple-choice items. The advantages to this approach are greater standardization of the assessment process, a more efficient and cost-effective administration and evaluation process, and a potentially higher degree of fidelity of the assessment itself. The downsides to the approach include less interpersonal contact with the applicant, inconclusive research on the validity of video/digital administration versus traditional paper and pencil administration, and the potential for the technology to fail during the test. unproctored Internet Testing. Unproctored Internet testing (UIT) is a more controversial procedure in assessment circles. Generally, when administering a job simulation (as well as other tests) in proctored testing, the applicant appears at a location, checks in with the administrator, presents identification, and is observed while completing the test. These procedures help assure the security and consistency of the testing procedures. In UIT, the test is placed (securely or non-securely) on an Internet site, and applicants complete the test in the setting of their choice (at home, at work, at a local library). Organizations in the private and Federal sectors have become more interested in this approach because it can save considerable resources in terms of the time, money, logistics, and staff needed for establishing testing sites.52 In addition, UIT can help reach and test a larger applicant pool, reduces applicant 52 MSPB did not ask about the use of UIT on the interrogatory sent to agencies, and therefore has no specific data on the extent to which it is used in the Federal Government. 36 Job Simulations: Trying Out For A Federal Job Formulating An Assessment Strategy time to devote to their duties. Because serving as an assessor is generally a collateral duty assignment, some assessors may find that other competing priorities prohibit them from efficiently carrying out their duties. Third, consider the diversity of the assessors. This diversity should take into consideration not only traditional diversity factors such as race, ethnicity, age, and gender, but also organizational level and functional work area.57 Finally, agencies should consider using managers and SMEs as assessors because they know the job. Some research indicates that if they are trained in proper assessment techniques, managers and SMEs may outperform even assessment professionals in evaluating candidates.58 At a minimum, assessors should have some type of subject matter expertise, insight into the organization’s needs and values, or unique skills in behavioral assessment.59 The four agencies highlighted in Appendix A of this report use agency managers or SMEs to assess applicants using simulations, and one agency even uses customers on its evaluation panels. Anyone responsible for either administering the test or rating the candidates should be well trained in how to carry out their responsibilities. The Assessment Center Guidelines, while providing guidance for one particular assessment type, contain some good points to consider when training assessors in general. First, agencies should ensure that the content of the training is appropriate. At a minimum, the training should help assessors understand the job, the dimensions being measured, the assessment techniques being used, and the evaluation procedures. The training length should be appropriate to: (1) the trainer and instructional design used; (2) the skill and knowledge levels of the assessors; and (3) the content and complexity of the job and assessment technique(s) being used.60 Agencies should keep in mind that when considering developing a new assessment strategy, such as using job simulations, they do not have to do an immediate overhaul of their entire program. That approach can be overwhelming. A formal restructuring of the assessment program can require a large commitment of resources to build and manage the program. This kind of support and commitment from agency leadership is not always available. Instead, selecting officials and HR staffs can collaboratively review the current process and identify small, informal steps they can take to improve the hiring process. For instance, they may find that the job analysis is out of date and needs to be updated before 57 Assessment Center Guidelines, Guidelines and Ethical Considerations for Assessment Center Operations, p. 4. 58 Caldwell, Thornton, and Gruys, “Ten Classic Assessment Center Errors: Challenges to Selection Validity,” p. 80. 59 Caldwell, Thornton, and Gruys, “Ten Classic Assessment Center Errors: Challenges to Selection Validity,” p. 80. 60 Assessment Center Guidelines, Guidelines and Ethical Considerations for Assessment Center Operations, pp. 6-7. A Report by the U.S. Merit Systems Protection Board 37 Formulating An Assessment Strategy the next hiring cycle. A review of the existing assessments may show that they are using too many or too few, or that they are just asking the wrong questions. Agencies may also want to consult with OPM, whose Products and Services group works with agencies on a reimbursable basis and may be able to help the agency assess its current situation and focus its efforts for little to no cost. As mentioned in the previous section, most of the organizations that reported using job simulations in the past year believed that the simulations increased the quality of the agency’s selections, but few reported actually measuring the impact. Such measurement is a critical step in the selection process. Measuring the costs, benefits, and results allows an organization to determine which assessments identify the best candidates and, therefore, where assessment funding and resources should be targeted. Further, such evaluations can help justify resource allocation to agency leadership, a critical step in this time of limited resources. The organizations that did measure the effectiveness of these assessments generally conducted validation studies, used surveys of HR and selecting officials, and measured the rate of successful probationary employees. These are valuable steps and indicate that there are different levels at which selection procedures should be measured. First, as discussed previously, a good assessment is reliable, valid, and fair. Each of these factors can, and should, be statistically measured to ensure that the assessments are meeting these criteria and are legally defensible if litigation is brought. In particular, the research shows that as the assessment’s predictive validity increases, so too does the percentage of the persons hired who become successful employees. Although, content valid assessments may be easier to develop and still legally defensible, criterion-related validity achieved through formal validity studies will help increase the organization’s return on investment for the assessment. Second, assessments can be measured in terms of participants’ perceptions. Good assessments should result in positive applicant reactions, so it may be beneficial to obtain applicant feedback. Selecting officials can provide feedback regarding their satisfaction with the process and the resulting new hires. Those involved in the selection process— e.g., the test developers, administrators, and assessors— can provide input on what went well, what did not, and what improvements should be considered. Finally, there are HR indicators that can demonstrate the success of the assessment process. Examining employee retention and the rate of successful probationary employees can tell the organization if the assessment resulted in a good person-to-job fit. Ultimately, the quality of the workforce and how well it is prepared to carry out (5) Measure the Success of the Assessment Procedures 38 Job Simulations: Trying Out For A Federal Job Formulating An Assessment Strategy the agency’s mission is the ultimate outcome of any human resources process, including assessment. Workforce quality is not an easy concept to measure, but demonstrating that the assessments resulted in a workforce that can efficiently and effectively perform the mission will solidify the business case for committing resources to assessment. Job simulation assessments can be an effective tool to evaluate applicant qualifications. They have many advantages, including relatively high levels of validity, better person- to-job fit because of the realistic job preview, a greater degree of fairness, and generally positive applicant perceptions. They do have their drawbacks, though, particularly the potentially high resource costs to develop and administer more advanced simulation exercises. However, the evidence does suggest that agencies could achieve a higher return on investment if they use more predictive assessments during the selection process, making job simulations a good option. Nonetheless, job simulations may not work in every situation. That is why it is important for agencies to have a good grasp of the job for which they are hiring, the competencies needed for that job, and knowledge about which assessments would best fit their specific needs and which would not help them. We have brought up a number of factors for agencies to consider when making decisions about their assessment process. They are by no means exhaustive. However, the five steps outlined in this assessment strategy should help get agencies on the right path to developing and implementing an assessment program that assists them in selecting employees on the basis of relative ability, knowledge, and skills, as set forth in the merit system principles. Conclusion A Report by the U.S. Merit Systems Protection Board 41 Appendix A in the study.63 However, more recent work in this area has pointed to the need for more research. A 2005 meta-analysis by Roth, Bobko, and McFarland pointed out limitations in the way work samples have been studied. Correcting for these perceived flaws, their study found the validity coefficient to be closer to .33, which is still better than many other assessment methods.64 Fairness. Assessment research generally considers work samples to have fairly high face validity, meaning that applicants view them as fair because they can see the relationship between the task performed in the work sample and the tasks to be performed on the job. One study found that work samples and structured interviews were viewed more favorably by applicants than other assessments.65 Typically, work samples have been viewed as having very low adverse impact, which means members of particular ethnic, racial, or gender groups are not disadvantaged by the nature of the assessment. However, this finding was recently challenged by a 2008 study which found that when controlling for perceived flaws in previous work sample studies, the work sample did result in higher black-white differences than had been previously thought.66 Again, more research in the area of work sample assessments could help resolve differing points of view in the research community. Though the jury is still out on the level of adverse impact, a recent study did show that work samples have a lower degree of exposure to discrimination lawsuits that are based on the selection procedure used. A review of Federal court cases involving hiring or selection discrimination from 1978 to 2000 found that work sample tests were less likely to be challenged in court than some other assessments (such as cognitive ability tests or unstructured interviews) and survived almost all of the litigation taken against them.67 Other Advantages. Because work samples ask applicants to actually perform a specific task that is necessary to do the job, applicants are less able to fake proficiency with the task and/or competencies required than other non-cognitive forms of assessment (e.g., personality tests). In addition, the work sample provides a realistic job preview of what the applicant will be doing on the job. If an applicant is able to determine if the job is a good fit for them before accepting the position, this could reduce turnover while increasing productivity and job satisfaction. 63 Schmidt and Hunter, “The Validity and Utility of Selection Methods in Personnel Psychology,” p. 265. 64 Philip L. Roth, Philip Bobko, and Lynn A. McFarland, “A Meta-Analysis of Work Sample Test Validity: Updating and Integrating Some Classic Literature,” Personnel Psychology, Winter 2005, Vol. 58, No. 4, p. 1009. 65 Hausknecht, Day, and Thomas, “Applicant Reactions to Selection Procedures.” 66 Philip Roth, Philip Bobko, Lynn McFarland, and Maury Buster, “Work Sample Tests in Personnel Selection: A Meta-Analysis of Black-White Differences in Overall and Exercise Scores,” Personnel Psychology, Autumn 2008, Vol. 61, No. 3, p. 637. 67 David E. Terpestra and R. Bryan Kethley, “Organizations’ Relative Degree of Exposure to Selection Discrimination Litigation,” Public Personnel Management, Fall 2002, Vol. 31, No. 3, pp. 288-289. 42 Job Simulations: Trying Out For A Federal Job Appendix A Other Disadvantages. While some work samples can be developed and administered quite easily, more complex tests can take a lot of time and resources to develop and administer. Often, they can only be administered to one applicant at a time or to a small group of applicants. Technology can help to reduce some of these requirements, but may only be able to provide limited relief. Work Samples in Action The Internal Revenue Service’s (IRS) Contact Representative (Customer Service Representative) is a mission-critical, entry-level position with no specific education or selective factor requirements. Contact Representatives are the people who answer tax questions for the public on the 1-800 number. This job is critical in ensuring that the public obtains the information it needs to correctly meet tax code requirements. To help identify high-quality applicants for this position at nationwide locations, IRS developed the Telephone Assessment Program (TAP). During the TAP, an applicant is seated at a desk with a telephone and given 30 minutes to review a packet of information that describes the job and explains the procedures used for the assessment. During this 30-minute period, the applicant can ask questions of the HR staff administering the assessment to make sure he or she understands the process. The applicant then receives four calls that are 5 to 7 minutes in length. Each call presents a scenario in which the applicant must carry out the duties of the position. The applicant can use the information packet to find the answers to each “taxpayer’s” questions. The “taxpayer” is a trained evaluator who is rating the applicant on seven competencies that were identified through a job analysis and working with subject matter experts and managers. For each competency, there is a set of benchmarks the evaluator uses to measure the applicant’s performance. The first call is a practice call, and the evaluator can come out of his role to assist the applicant if necessary. The three other role-plays are rated against the benchmarks. The evaluator records observations/ratings on a computer screen as they go through the call. The total time of the assessment is about 1 hour. The assessment was developed by IRS senior specialists and I/O psychologists with the assistance of a contractor. They conducted a validation study that showed a strong relationship between the assessment and success on the job. They have looked at adverse impact and found virtually none. The assessment has been updated periodically for content. The IRS had top management support and dedicated resources for improving assessments from the beginning. In 1998, Congress passed legislation to modernize IRS. As part of that effort, the Commissioner wanted a more professional workforce, more reliance on technology, and to reduce the burden on managers. He also wanted to explore new and innovative ways to carry out the mission. A Report by the U.S. Merit Systems Protection Board 43 Appendix A Both applicant and supervisor reactions have been positive. The IRS has found that the work sample assessment provides a realistic job preview that gives applicants the opportunity to experience both the positive and negative aspects of the job before accepting a job offer. Anecdotal evidence suggests that the assessment has helped reduce turnover. The key drawbacks to the assessment are its expense and resource requirements. The IRS is currently piloting a new process that integrates online assessment with the TAP to help reduce resource and applicant burden. Definition. A situational judgment test presents applicants with a description of a work-related problem scenario and asks them to exercise their judgment by choosing or evaluating alternative courses of action to the situation. The tests are almost always multi-dimensional and test different skills and abilities. Historically, SJTs have been paper and pencil tests, making them a low fidelity simulation. However, there is a growing trend in video-based testing which uses video technology to present the scenarios and even to record applicants’ responses. The technology provides applicants with a more realistic feel and greater job preview. Appropriate Uses. SJTs have been found to be effective measures of social functioning dimensions such as conflict management, interpersonal skills, problem solving, negotiating, and teamwork. They have also been found to be particularly useful for assessing managerial and leadership competencies.68 They are widely used by public organizations, particularly for law enforcement and firefighter positions as well as other occupations that require high levels of independent judgment. Examples. Typically, SJT prompts are either behavioral or knowledge-based.69 Behavioral tests ask respondents how they would personally react to a given situation, as shown in the following example of a multiple-choice question: You have noticed that there is a lack of cooperation and trust among the employees in the organization you manage. This counter-productive behavior is adversely affecting both the quality and the quantity of the work produced by the employees. You would most likely— 68 Office of Personnel Management, “Section III: Situational Judgment Tests,” Assessment Decision Guide, http://apps.opm.gov/ADT/. 69 Michael A. McDaniel, Nathan S. Hartman, Deborah L. Whetzel, and W. Lee Grubb III, “Situational Judgment Tests, Response Instructions, and Validity: A Meta-Analysis,” Personnel Psychology, Spring 2007, Vol. 60, No. 1, p. 63. Situational Judgment Tests 46 Job Simulations: Trying Out For A Federal Job Appendix A showing lower levels of black-white subgroup differences than the paper and pencil SJT.73 Video-based tests may be more expensive to develop than pencil and paper tests, but they are often less expensive to develop than high fidelity simulations, such as work samples and assessment centers. Also, they can still be used for a large number of applicants in a single session, provide a realistic job preview, and have low levels of adverse impact. Situational Judgment Tests in Action The U.S. Customs and Border Protection (CBP) uses a Video-Based Test (VBT) for CBP officers and agricultural specialists that could be considered an advanced version of a situational judgment test that increases the fidelity of the assessment, provides a more realistic job preview, and reduces the burden on managers assessing the candidates’ responses. Through several critical incident job analyses, the Personnel Research Assessment Division (PRAD) identified a set of key tasks, competencies, and KSAs necessary to successfully perform the job. Specifically, the assessment concentrates on interpersonal, oral, and decision-making skills, which are competencies for which SJTs are an effective measurement. The I/O psychologist staff worked with subject matter experts to identify critical incidents to use in video-based scenarios. Applicants report to one of 50 VBT sites across the country. During the VBT, applicants are given instructions by the administrator as part of the narration in the video. There are eight scenarios that are each a few minutes long. The scenarios go from least difficult to most difficult. Applicants watch a job-related situation on a television monitor in a specially equipped testing room. Each scenario is followed by a 45-second response period in which applicants act out what they would do in response to the scenario presented. The role-play response is captured on video. To get comfortable with this interactive type of role-play, CBP allows two test scenarios. In total, the VBT lasts about 30 minutes per applicant. The videos can then be sent to over 300 ports of entry to be rated by trained raters— generally supervisory CBP officers who were chosen based on a list of criteria and who receive 1.5 days of training in the assessment and how to evaluate responses. When conducting the ratings, the supervisory CBP officers can fast-forward through the instructions and scenarios and focus on viewing the applicant responses and providing ratings, reducing the amount of time it takes to assess an applicant down to about 10 minutes. In addition, the tape-recorded applicant responses are saved and used as documentation, which reduces the amount of note-taking that raters must complete. Furthermore, the VBT can be rated at any time, such as during non-peak hours, reducing the operational burden. 73 David Chan and Neal Schmitt, “Video-Based Versus Paper-and-Pencil Method of Assessment in Situation Judgment Tests: Subgroup Differences in Test Performance and Face Validity Perceptions,” Journal of Applied Psychology, 1997, Vol. 82, No. 1, p. 143. A Report by the U.S. Merit Systems Protection Board 47 Appendix A The VBT is used as part of a multiple hurdle process. Applicants are given the opportunity to voice their opinion about the process through a short survey given upon completion of the testing session, and responses indicate that the assessment is well received. It has also been well received by selecting officials. The VBT has been so successful for CBP hiring that its use has been extended to the agriculture specialist position. PRAD received the 2007 International Public Management Association for Human Resources Assessment Council (IPMAAC) Innovations in Assessment Award in recognition of its work in developing and implementing the Video-Based Test. Definition. Assessment centers evaluate applicants on their job-related KSAs using multiple, standardized exercises. Each exercise is developed to measure clearly defined dimensions of behavior. Dimensions are clusters of activities important to the job that are observable, are specific, and consist of tasks related to the job. Trained assessors use predetermined criteria to systematically score the applicants’ performance on each exercise. The scores on each of the assessments are statistically integrated so that each exercise contributes to the applicant’s overall score (as opposed to a multiple hurdle process in which each exercise is used to screen applicants out of the successive exercises). There is no set number of exercises used during this process, but the International Task Force on Assessment Center Guidelines stipulates that at least one of the assessments must be a simulation, defined by the Task Force as “an exercise or technique designed to elicit behaviors related to dimensions of performance on the job requiring the participants to respond behaviorally to situational stimuli.”74 Assessment centers were first used by military organizations and intelligence services to select officers and undercover agents during World War II. The skills necessary for these roles were rather complex and not easily measured by existing tools. Then, in the 1950s, Douglas Bray of AT&T began applying the technique to the business environment. This resulted in an extremely detailed analysis of assessment centers and their capability to measure success on the job. AT&T found a strong correlation between the assessment center scores of more than 400 entry-level employees and the different measures of managerial progress. A follow up study was conducted in the 1970s. Ultimately, these groundbreaking studies found that management potential is highly predictable using assessment center techniques. Appropriate Uses. Research has found that assessment centers can be effectively used for both employee selection and career development. They have been cited as being particularly helpful in selecting or developing supervisors, managers, and executives for promotional opportunities or development programs. 74 Assessment Center Guidelines, Guidelines and Ethical Considerations for Assessment Center Operations, p. 3. Assessment Centers 48 Job Simulations: Trying Out For A Federal Job Appendix A Assessment centers require considerable time and financial resources. They take time and expertise to develop and organize. They require multiple trained raters. They require space, equipment, and materials to administer. All of these resources amount to a fairly significant cost. Because of the resource requirements, this type of assessment is best used for mission-critical and/or high volume occupations. For that reason, large organizations tend to use them more than small organizations. While assessment centers are resource intensive, they can be streamlined by using technology that can reduce the cost and resources needed to administer them. Some organizations have started using online exercises. Some organizations have also turned to video technology to administer and record simulation exercises. Examples. Assessment centers always use more than one exercise to measure the different applicant dimensions, such as group, individual, written, analytical, and/or role-play exercises. While there are many different exercises that can be used, both simulation and non-simulation, some are more common than others. For instance, a study found that: 75 • 83 percent of ACs used in-basket exercises, which assess an applicant’s ability to perform managerial administrative duties. The applicant is instructed to take action on various issues or problems that have accumulated in a manager’s in- basket. The in-basket may contain letters, reports, telephone messages, emails, or other communications from supervisors or staff that need to be addressed, and the applicant’s actions are evaluated using predefined criteria. • 78 percent used simulated coaching meetings. During this simulated exercise, applicants role-play situations in which they are resolving performance issues with a subordinate. • 70 percent used leaderless group discussions (LGDs), a simulation exercise in which applicants are presented with an issue or problem and put into groups to develop solutions. For instance, the group may be instructed to develop a policy or to work out budget issues among their divisions. The applicants’ roles may be defined or undefined and they may be put in situations of collaboration or competition. All applicants are separately rated on their participation and response to the situation. • 70 percent used structured interviews. Structured interviews are interviews in which the applicants are asked the same questions in the same order and are rated on their responses using the same rating scale. Guidelines recommend that at least two exercises be used to measure each competency dimension. Some organizations use off-the-shelf exercises for their assessment centers, but this is generally not recommended. The exercises should be directly tied to the job analysis and be tailored to the job for which the organization is hiring to be most effective. 75 Warren Bobrow, “Assessment Centers,” Intervention Resources Guide: 50 Performance Improvement Tools, Danny G. Langdon, Kathleen S. Whiteside, and Monica M. McKenna, eds., Jossey-Bass/Wiley: San Francisco, CA 1999. A Report by the U.S. Merit Systems Protection Board 51 Appendix A The result of the assessment center is a registry that is rank-ordered by the scores obtained during the assessment battery. Ultimately, FCS places about 40 of the top performers, for a total cost of about $12,000 per successful offer. Though the process is costly, FCS has been very happy with the quality of the applicants and believes it is getting a good return on its investment. Definition. In a job tryout, applicants are generally hired with minimal screening of their qualifications and are given an evaluation period. During the evaluation period, they receive the training they need to perform the duties of the job, and their performance is evaluated to determine if they meet the established levels of satisfactory performance. At the end of the evaluation period, they are either retained or terminated based on their performance. There is not a significant amount of professional literature pertaining to job tryout procedures. This is probably because true job tryouts are not used very often. Most organizations would view them as impractical because of the high cost of terminating lightly screened, low performers. Appropriate Uses. The job tryout procedure can be used with almost any kind of position, particularly entry-level jobs. The evaluation period may vary, depending on the complexity of the job. Typically, it is about 6 to 8 months.80 An easy-to-learn position may have a short evaluation period, such as two weeks, whereas a more complex position could have a longer evaluation period. However, users of this procedure should keep in mind that the more training that is required for the position, the lower the return on investment will be because more resources will be put into an employee who was minimally screened and could easily fail. Also, job tryouts may be more useful for high- volume occupations with high turnover that need applicants continuously in the pipeline. The organization would get a higher return on investment by using its resources to train and assess a larger volume of people that can fill multiple vacancies than by using those same resources to train and assess applicants for a single position. Examples. In the Federal Government, new employees are generally subject to a probationary or trial period. The purpose of the probationary or trial period is to provide the Government the opportunity to evaluate the employee on the job to determine if the appointment should become final at the end of the period. Until the probationary period has been completed, a probationer is still just an applicant for an appointment. If used correctly, the probationary or trial period could be considered a job tryout procedure. 80 Schmidt and Hunter, “The Validity and Utility of Selection Methods in Personnel Psychology,” p. 268. Job Tryout Procedures 52 Job Simulations: Trying Out For A Federal Job Appendix A Scoring. A true job tryout procedure will have established criteria defining what satisfactory performance on the job is. The criteria will be based on a job analysis that defines what the duties and tasks of the position are and what outcomes are expected from successful completion of those duties and tasks. The people hired into the program will be trained in the duties and tasks necessary to complete the job, will be told how they will be evaluated throughout the tryout, and will be evaluated by a trained assessor/ supervisor against the pre-established criteria. Those who do not meet the expected performance by the end of the evaluation period will be terminated. Validity. The two key studies that address job tryout procedures found them to be highly predictive of job performance, with a validity coefficient of .44. Some may be surprised that the coefficient is not higher, given that the “applicant” actually performs the job for a significant period of time. However, the study authors found that supervisors are reluctant to terminate marginal performers. Because this is an unpleasant action, supervisors are more likely to lower the standards by which they evaluate the applicants, thereby undermining the effectiveness of the assessment.81 Our research on the Federal probationary period supports this finding, given that many supervisors we surveyed did not intend to remove probationers who they thought were not an asset to the organization.82 Thus, the key to ensuring an effective, predictive job tryout is following through on terminating employees who do not meet the standards for satisfactory performance. Fairness. We could not find any in-depth research concerning the fairness and/or adverse impact of job tryouts. It would seem that this type of simulation assessment would be considered fair and objective, simply because the applicants are actually hired into the position and given the opportunity to succeed. However, we know that Federal probationers do raise claims of discrimination, so we need to be cognizant that fairness issues may exist. Job Tryouts in Action Best Friends Animal Society is a nonprofit organization that operates the Nation’s largest sanctuary for abused and neglected animals. The sanctuary is located in a small, remote town in Utah, and the work performed at the sanctuary can be both physically and emotionally grueling. Both of these factors have led to problems with employee turnover and job burnout. To ensure that new employees are able to adapt to the small town life and can endure the physical and emotional demands of the job, Best Friends uses a job tryout procedure to assess applicants for many of their entry-level positions, including animal caregivers and support staff. 81 Schmidt and Hunter, “The Validity and Utility of Selection Methods in Personnel Psychology,” p. 268; Hunter and Hunter, “Validity and Utility of Alternative Predictors of Job Performance,” p. 82. 82 MSPB, The Probationary Period: A Critical Assessment Opportunity, p. 7. A Report by the U.S. Merit Systems Protection Board 53 Appendix A The hiring process starts with the applicant submitting an application on-line. The HR staff reviews the applications, conducts telephone interviews, and works with the hiring manager to determine who will be invited for the 2-week evaluation period. They extend the invitation to the applicant and schedule the evaluation period generally within 2 to 4 weeks of the notification. Applicants use their own time and pay their own expenses (including accommodations, transportation, and meals) to come for the 2-week job tryout. Best Friends pays each applicant $10 per hour and does not guarantee employment at the end of the tryout. Because their mission is so appealing to people in the industry, applicants generally do not balk at the job tryout requirements. Best Friends has established a cadre of trainers who provide applicants with the necessary training, give them their assignments, and evaluate them as they perform their duties. They have identified a set of core competencies for the occupations being evaluated and established legally defensible benchmarks against which applicants are measured. The trainers record their observations and feedback and provide these to the hiring manager. The hiring manager meets with the applicant at the beginning, middle, and end of the 2-week evaluation period to provide the applicant with feedback. About 80 percent of those who make it to the job tryout are hired. Reaction from the applicants has been very good. They understand the need for the evaluation period and are supportive of it. There have only been a few instances in which applicants were not able to attend the tryout because of logistics, and these applicants were kept in the pipeline for future positions. The 2-week paid job tryout has been in place for almost 10 years, and Best Friends reports that it has been successful in helping the organization hire a qualified workforce that can deal with the unique location and work of the sanctuary. 56 Job Simulations: Trying Out For A Federal Job Appendix B Agency/subagency: ____________________________________________________ Contact name/title: ___________________________________________________ Contact organization: __________________________________________________ Contact telephone: ____________________________________________________ Contact email: _______________________________________________________ 1. Overall, how would you rate the quality of the applicants for your agency/ subcomponent’s vacancies? — Excellent — Very Good — Good — Fair — Poor 2. Overall, how would you rate the quality of the new employees selected by your agency/subcomponent? — Excellent — Very Good — Good — Fair — Poor 3. Overall, how knowledgeable would you say the following are about the assessment tools available for measuring applicant qualifications? Agency Selecting Officials: — Very knowledgeable — Knowledgeable — Somewhat knowledgeable — Not very knowledgeable Agency Human Resources Staff: — Very knowledgeable — Knowledgeable — Somewhat knowledgeable — Not very knowledgeable 4. Who is typically responsible for deciding what assessment tools will be used to measure applicant qualifications? (select all that apply) — Agency HR staff — Agency Industrial/Organizational (IO) Psychologist or other testing staff — Selecting official(s) — Subject matter expert(s) or line staff — Office of Personnel Management — Other (please specify) Questionnaire: A Report by the U.S. Merit Systems Protection Board 57 Appendix B 5. If you marked the selecting official or subject matter expert above, how much assistance/advice does the HR office provide regarding the value of different assessment tools. — A Lot (e.g, HR consults regularly with selecting officials about their options) — Some (e.g., HR provides assistance when selecting official have questions) — Little (e.g., HR does not have the resources to provide regular consultation) 6. In the past year, which of the following assessment methods has your agency/ subcomponent used to assess applicant qualifications? (select all that apply) — Structured interviews that employ objective, pre-determined rules for eliciting, observing, and evaluating interview responses — unstructured interviews in which questions asked may be unplanned and vary across interviews, and the results are analyzed and applied subjectively — Cognitive ability tests that estimate applicants’ abilities involved in thinking (e.g., reasoning, perception, memory, verbal and math ability, and problem solving) — Ratings of education and experience as described through occupational questionnaires (e.g., QuickHire, AVUE, or USAStaffing multiple-choice questions) — Ratings of education and experience as described in kSA narratives (knowledge, skills, and abilities) — Reference checks that evaluate an applicant’s past job performance using information collected from those who have prior knowledge of the applicant — Biographical data that includes questions about past events and behaviors ref lecting personality attributes, attitudes, experiences, interests, skills and abilities validated as predictors of overall performance for a given occupation (e.g., ACWA Form C) — Accomplishment records in which applicants provide written descriptions of personal accomplishments that best illustrate their proficiency on critical job competencies, and a panel of trained raters use competency-based benchmarks to score the degree to which the behaviors and outcomes ref lect the benchmark levels of proficiency — Job knowledge tests that measure the applicant’s current knowledge of the field/job (e.g., a test that measures an applicant’s knowledge of basic chemistry or accounting principles). — Grade point average (e.g., Outstanding Scholar requirement) — Education level (e.g., college graduate, graduate degree, Ph.D.) 58 Job Simulations: Trying Out For A Federal Job Appendix B 7. In the past year, which of the following types of job simulations has your agency/ subcomponent used to assess applicant qualifications? (select all that apply) — Work sample tests that require the applicant to actually perform a critical job task to measure his/her current ability to perform a part of the job (e.g., a role-play between a customer service representative applicant and an angry customer or an x-ray technician reading an x-ray). — Situational judgment tests that measure how an applicant might react to a particular situation (e.g., showing the applicant a video of a specific situation and asking them a variety of questions that would demonstrate how they might react in that situation). — Assessment centers in which applicants’ performance is observed in several job-related exercises, such as leaderless group discussions and business games (e.g., the assessment procedure for Presidential Management Fellows). — Job try-out procedures in which an applicant is hired with minimal screening, evaluated over a specified period of time to determine if he/she is a good fit for the job, and is actively separated or retained on the basis of performance (as opposed to a typical probationary or trial period in which finalizing the applicant’s appointment is fairly standard procedure). — We have not used job simulations in the past year (Go to Question 20) — Other type(s) of job simulation. Please list. For agencies that have used a job simulation in the past year, please answer the following questions in regard to the simulations administered in the past year. For agencies that have not used a job simulation in the past year, please go to Question 20. 8. For approximately what percentage of your organization’s individual vacancies has a job simulation assessment been used in the past year (count open continuous vacancies by the number of times closed during the year and include Presidential Management Fellow selections)? — Less than 10% — 11-25% — 26-50% — 51-75% — More than 75% 9. For what types of positions does your organization use job simulations more frequently? — Merit Promotion — Delegated Examining — Both equally A Report by the U.S. Merit Systems Protection Board 61 Appendix B 19. Please describe briefly how your agency determines if it will use a job simulation to assess applicants. 20. MSPB would like to highlight best practices in the area of job simulations. Please describe any job simulation methods you use that you feel have been especially effective. 21. What obstacles have you faced in using job simulations OR what factors have kept you from using job simulations? (select all that apply) — The agency’s jobs do not lend themselves well to this type of assessment — We use other assessments that we feel better differentiate among candidates — The agency does not have the resources to develop and/or administer — The agency does not have the expertise to develop and/or administer — The agency does not have the time required to develop assessments — The agency does not have the time required to train those who administer the assessments — The HR staff does not have enough knowledge about these types of assessments to determine if they would be beneficial — Selecting officials do not have enough knowledge about these types of assessments to determine if they would be beneficial — Other (please list) Thank you for your participation! You may email your response to laura.shugrue@mspb.gov; fax it to (202) 653-7211; or mail it to U.S. Merit Systems Protection Board, Office of Policy and Evaluation, ATTN: Laura Shugrue, 1615 M Street, NW, Washington, DC 20036 U.S. Merit Systems Protection Board 1615 M Street, N.W. Washington, DC 20419 Official Business Penalty For Private Use $300 PRSRT STD U.S. POSTAGE PAID WASHINGTON DC PERMIT NO. G-113 JOB SIMULATIONS: TRYING OUT FOR A FEDERAL JOB