Critical Incidents in Psychology and Aviation: Collection, Analysis, and Applications, Exams of Latin language

Download Critical Incidents in Psychology and Aviation: Collection, Analysis, and Applications and more Exams Latin language in PDF only on Docsity! VOL. 51, No. 4 JULY, 1954 Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE JOHN C. FLANAGAN American Institute for Research and University of Pittsburgh During the past ten years the writer and various collaborators have been engaged in developing and utilizing a method that has been named the "critical incident technique." It is the purpose of this article to describe the development of this methodology, its fundamental principles, and its present status. In addition, the findings of a considerable number of studies making use of the critical incident technique will be briefly reviewed and certain possible further uses of the technique will be indicated. The critical incident technique consists of a set of procedures for collecting direct observations of human behavior in such a way as to facilitate their potential usefulness in solving practical problems and developing broad psychological principles. The critical incident technique outlines procedures for collecting observed incidents having special significance and meeting systematically defined criteria. By an incident is meant any observable human activity that is sufficiently complete in itself to permit inferences and predictions to be made about the person performing the act. To be critical, an incident must occur in a situation where the purpose or intent of the act seems fairly clear to the observer and where its consequences are sufficiently definite to leave little doubt concerning its effects. Certainly in its broad outlines and basic approach the critical incident technique has very little which is new about it. People have been making observations on other people for centuries. The work of many of the great writers of the past indicates that they were keen observers of their fellow men. Some of these writers must have relied on detailed notes made from their observations. Others may have had unusual abilities to reconstruct memory images in vivid detail. Some may have even made a series of relatively systematic observations on many instances of a particular type of behavior. Perhaps what is most conspicuously needed to supplement these activities is a set of procedures for analyzing and synthesizing such observations into a number of relationships that can be tested by making additional observations under more carefully controlled conditions. BACKGROUND AND EARLY DEVELOPMENTS The roots of the present procedures can be traced back directly to the studies of Sir Francis Galton nearly 70 years ago, and to later developments such as time sampling studies of recreational activities, controlled observation tests, and anecdotal records. The critical incident technique as such, however, can best be regarded as an outgrowth of studies in the Aviation Psychology Program of the United States Army Air Forces in World War II. The Aviation Psychology Program was established in the summer of 1941 to develop procedures for the selection and classification of aircrews. One of the first studies (40) carried out in this program was the analysis of the specific reasons for failure in learning to fly that were reported for 1,000 pilot candidates eliminated from flight training schools in the summer and early fall of 1941. The basic source used in this analysis was the proceedings of the elimination boards. In these proceedings the pilot instructors and check pilots reported their reasons for eliminating the particular pilot. It was found that many of the reasons given were clichés and stereotypes such as "lack of inherent flying ability" and ''inadequate sense of sustentation," or generalizations such as ''unsuitable temperament," "poor judgment," or "insufficient progress." However, along with these a number of specific observations of particular behaviors were reported. This study provided the basis for the research program on selecting pilots. Although it was found very useful, it also indicated very clearly the need for better procedures for obtaining a representative sample of factual incidents regarding pilot performance. A second study (13), which emphasized the importance of factual reports on performance made by competent observers, was carried out in the winter of 1943-1944 in the 8th, 9th, 12th, and 15th Air Forces. This study collected the reasons for the failures of bombing missions as reported in the Group Mission Reports. Although in the preparation of these reports much greater emphasis was given to determining the precise facts in the case, it was apparent that in many instances the official reports did not provide a complete record of all the important events. Even with these limitations, the information given was found to be of considerable value, and the systematic tabulations that were prepared provided the basis for a series of recommendations that resulted in important changes in Air Force selection and training procedures. In the summer of 1944 a series of studies (74) was planned on the problem of combat leadership in the United States Army Air Forces. These represent the first large-scale, systematic effort to gather specific incidents of effective or ineffective behavior with respect to a designated activity. The instructions asked the combat veterans to report incidents observed by them that involved behavior which was especially helpful or inadequate in accomplishing the assigned mission. The statement finished with the request, "Describe the officer's action. What did he do?" Several thousand incidents were collected in this way and analyzed to provide a relatively objective and factual definition of effective combat leadership. The resulting set of descriptive categories was called the "critical requirements" of combat leadership. variations in flying conditions in the types of incidents reported. The study also demonstrated the selective recall of dramatic or other special types of incidents. This bias was especially noticeable in the incidents reported several months after their occurrence. The incidents obtained in this study were used to develop procedures for evaluating the proficiency of air route traffic controllers and also for developing a battery of selection tests for this type of personnel. In the spring of 1949 the American Institute for Research undertook a study to determine the critical job requirements for the hourly wage employees in the Delco-Remy Division of the General Motors Corporation. This study, reported by Miller and Flanagan (46), was the first application of these techniques in an industrial situation. Foremen who were members of a committee appointed to develop employee evaluation procedures collected 2,500 critical incidents in interviews with the other foremen in the plants. On the basis of these data a form was prepared for collecting incidents on a day-to-day basis as a continuous record of job performance. Using this form, the Performance Record for Hourly Wage Employees (21), three groups of foremen kept records on the performance of their employees for a two-week period. A group of 24 foremen recorded incidents daily; another group of 24 foremen reported incidents at the end of each week; and a third group containing the same number of foremen reported incidents only at the end of the two-week period. The three groups of foremen represented comparable conditions of work and supervision. The foremen reporting daily reported 315 critical incidents; the foremen reporting weekly, 155 incidents; and the foremen reporting only once at the end of two weeks reported 63 incidents. Thus, foremen who reported only at the end of the week had forgotten approximately one half of the incidents they would have reported under a daily reporting plan. The foremen who reported only at the end of the two-week period appeared to have forgotten 80 per cent of the incidents observed. Although it is possible that the find- ings may be partially attributed to the fact that the foremen making daily records actually observed more critical incidents because of the daily reminder at the time of recording, it is clear that much better results can be expected when daily recording is used. Another analysis based on data collected at the Delco-Remy Division compared the number of critical incidents of various types obtained from interviews with those recorded daily by the foremen on the performance record. Although there were some differences in the relative frequencies for specific categories, the general patterns appeared to be quite similar. These results suggest that critical incidents obtained from in- terviews can be relied on to provide a relatively accurate account of job performance if suitable precautions are taken to prevent systematic bias. In addition to the development of the performance record described above, the critical incidents collected in this study were used as the basis for constructing selection tests covering both aptitude (18) and attitude (2) factors. STUDIES CARRIED OUT AT THE UNIVERSITY OF PITTSBURGH A substantial number of studies have been carried out in the department of psychology at the University of Pittsburgh by students working for advanced degrees under the author's direction. Most of these studies had as their objective the determination of the critical requirements for a specific occupational group or activity. Many of them also included contributions to technique. In 1949 Wagner (66) completed a dissertation on the critical requirements for dentists. In this study, critical incidents were obtained from three sources: patients, dentists, and dental school instructors. The incidents were classified into four main aspects of the dentist's job: (a) demonstrating technical proficiency; (b) handling patient relationships; (c) accepting professional responsibility; and (d) accepting personal responsibility. As might be expected, the patients did not report as large a proportion of incidents for demonstrating technical proficiency or accepting professional responsibility as did the other two groups, and the instructors reported only a relatively small proportion of their incidents in the area of handling patient relationships. On the basis of the findings from this study, a battery of selection tests was developed for use by the University of Pittsburgh School of Dentistry. A number of proficiency tests for measuring ability with respect to certain of the critical requirements were also developed using these results as a basis. Another dissertation completed in 1949 was Finkle's (11) study of the critical requirements of industrial foremen. This study was conducted in the East Pittsburgh plant of the Westinghouse Electric Corporation. Critical incidents were obtained from foremen, general foremen, and staff personnel. A number of points pertaining to technique were studied. One finding was in reference to the effect on the types of incidents obtained of the degree of importance or exceptionalness set up as a criterion for reporting or ignoring incidents. The incidents obtained from the use of questions that asked for incidents only slightly removed from the norm were compared with incidents obtained from questions intended to elicit more definitely effective or ineffective behaviors. Some examples of these questions are: 1. Think of a time when a foreman has done something that you felt should be encouraged because it seemed to be in your opinion an example of good foremanship. (Effective— slight deviation from norm.) 2. Think of a time when a foreman did something that you thought was not up to par. (Ineffective—slight deviation from norm.) 3. Think of a time when a foreman has, in your opinion, shown definitely good foreman- ship—the type of action that points out the superior foreman.(Effective—substantial de- viation from the norm.) 4. Think of a time when a foreman has, in your opinion, shown poor foremanship—the sort of action which if repeated would indicate that the man was not an effective foreman.(Ineffective—substantial deviation from norm.) The frequencies of incidents obtained in each of the 40 categories into which the effective behaviors were classified were compared for the questions requesting slight and substantial deviations from the norm, and the significance of the differences was tested by means of the chi-square test. Two of the differences were significant at the 1 per cent level and one at the 5 per cent level. Comparisons of the frequencies in each of the 40 categories for ineffective incidents failed to reveal any chi squares significant at either the 5 per cent or the 1 per cent level. The questions involving only a slight deviation from the norm resulted in more effective incidents concerned with gaining the respect and loyalty of the workers and also in more incidents that involved making, encouraging, and accepting suggestions. They produced significantly fewer incidents regarding fitting men to jobs. The small number of significant differences—only three in 80 comparisons—suggests that the types of incidents obtained are not very greatly changed by variations in wording of the questions comparable to those shown above. It seems likely that this is at least partially due to the fact that the persons interviewed report only incidents that represent a fairly substantial deviation from the norm regardless of the precise wording of the question asked. Another comparison made in this study related to the influence of asking for an effective or an ineffective incident first. About 10 per cent more incidents were obtained from booklets requesting effective incidents first than from booklets requesting in- effective incidents first. This difference was sufficiently small so that it could reasonably be attributed to chance sampling fluctuations. The incidents collected in this study were used, along with other data, in the preparation of a Performance Record for Foremen and Supervisors (23). A study was conducted by Nevins (50) on the critical requirements of bookkeepers in sales companies. She collected incidents relating to applicants for bookkeeping positions as well as for employees working in this capacity. For the collection of the information about the practicing bookkeepers, a modification in the critical incident technique was made. This was done because, in the bookkeeping profession, success and failure are usually defined in terms of persistent behavioral patterns. Occasional mistakes in adding and balancing accounts are expected, but repeated errors are considered serious. Instead of the single incident, therefore, many of the items included represented either a pattern of behaviors or a series of similar behaviors. Weislogel (72) determined the critical requirements for life insurance agency heads. A principal feature of his study related to the comparison of two types of agency heads— managers and general agents. It was believed that the critical behaviors for one type of agency head might provide a different pattern than that obtained for the other. This hypothesis was not confirmed by the analysis of the obtained incidents. The patterns of critical requirements were found to be quite similar for the two types of administrators. Smit (58) carried out a study to determine the critical requirements for instructors of general psychology courses. Perhaps the finding of most general importance in this study was the existence of substantial differences between the patterns of critical incidents reported by students and faculty. The faculty reported a significantly larger percentage of effective behaviors in the following areas: giving demonstrations or experiments, using discussion group techniques, encouraging and ascertaining students' ideas and opinions. The students, on the other hand, contributed a larger percentage of behaviors in the following areas: reviewing examinations, distributing grades, and explaining grades; using lecture aids such as drawings, charts, movies, models, and apparatus; using project techniques; giving test questions on assigned material; helping students after class and during class recess; the manner of the instructor. The faculty reported a larger percentage of ineffective behaviors concerning maintaining order. The ineffective behaviors that were reported in a larger percentage by students involved these areas: presenting requirements of the course, using effective methods of expression, dealing with students' questions, pointing out fallacies, reviewing and summarizing basic facts and principles, using project techniques, using verbal diag- bility at the present stage, in addition to examples of present best practice, the underlying principles for the step will be discussed and also the chief limitations with, wherever possible, suggestions for studies that may result in future improvements in the methods. 1. General Aims A basic condition necessary for any work on the formulation of a functional description of an activity is a fundamental orientation in terms of the general aims of the activity. No planning and no evaluation of specific behaviors are possible without a general statement of objectives. The trend in the scientific field toward operational statements has led a number of writers to try to describe activities or functions in terms of the acts or operations performed, the materials acted on, the situations involved, the results or products, and the relative importance of various acts and results. These analyses have been helpful in emphasizing the need for more specific and detailed descriptions of the requirements of activities. Typically, however, such discussions have failed to emphasize the dominant role of the general aim in formulating a description of successful behavior or adjustment in a particular situation. In its simplest form, the functional description of an activity specifies precisely what it is necessary to do and not to do if participation in the activity is to be judged successful or effective. It is clearly impossible to report that a person has been either effective or ineffective in a particular activity by performing a specific act unless we know what he is expected to accomplish. For example, a supervisor's action in releasing a key worker for a half a day to participate in a recreational activity might be evaluated as very effective if the general aim of the foreman was to get along well with the employees under him. On the other hand, this same action might be evaluated as ineffective if the primary general aim is the immediate production of materials or services. In the case of the usual vocational activities the supervisors can be expected to supply this orientation. In certain other types of activities, such as civic, social, and recreational activities, there frequently is no supervisor. The objectives of participation in the activity must then be determined from the participants themselves. In some instances, these may not be verbalized to a sufficient extent to make it possible to obtain them directly. Unfortunately, in most situations there is no one general aim which is the correct one. Similarly, there is rarely one person or group of persons who constitute an absolute, authoritative source on the general aim of the activity. In a typical manufacturing organization the foreman, the plant manager, the president, and the stockholders might define the general aim of the workers in a particular section somewhat differently. It is not possible to say that one of these groups knows the correct general aim and the others are wrong. This does not mean that one general aim is as good as another and that it is unimportant how we define the purpose of the activity. It does mean that we cannot hope to get a completely objective and acceptable general aim for a specific activity. The principal criterion in formulating procedures for establishing the general aim of the activity should be the proposed use of the functional description of the activity which is being formulated. Unless the general aim used is acceptable to the potential users of the detailed statement of requirements, the whole effort in formulating this statement will have been wasted. The most useful statements of aims seem to center around some simple phrase or catchword which is slogan-like in character. Such words provide a maximum of communication with only a minimum of possible misinterpretation. Such words as "appreciation," "efficiency," "development," "production," and "service" are likely to be prominent in statements of general aims. For example, the general aim of a teacher in elementary school art classes might be the development of an appreciation of various visual art forms on the part of the students. The general aim of the good citizen might be taken as effective participation in the development and application of the rules and procedures by which individuals and groups are assisted in achieving their various goals. With the aid of a form of the type shown in Fig. 1, the ideas of a number of well- qualified authorities can be collected. It is expected that in response to the question on the primary purpose of the activity many persons will give a fairly lengthy and detailed statement. The request to summarize is expected to get them to condense this into a brief usable statement. These should be pooled a a trial form of the statement of gener aim developed. This statemen should be referred either to these authorities or to others to obtain a final statement of the general aim tha is acceptable to them. Necessary revisions should be made as indicated by these discussions. Usually considerable effort is required to a defeating the purpose of the genera aim by cluttering up the statemen with specific details and qualifying conditions. nd al t t void l t In summary, the general aim of an activity should be a brief statement obtained from the authorities in the field which expresses in simple terms those objectives to which most people would agree. Unless a brief, simple statement has been obtained, it will be difficult to get agreement among the authorities. Also it will be much harder to convey a uniform idea to the participants. This latter group will get an over-all impression and this should be as close to the desired general aim as possible. 2. Plans and Specifications To focus attention on those aspects of behavior which are believed to be crucial in formulating a functional description of the activity, precise instructions must be given to the observers. It is necessary that these instructions be as specific as possible with respect to the standards to be used in evaluation and classification. The group to be studied also needs to be specified. One practical device for obtaining specific data is to obtain records of "critical incidents" observed by the reporting personnel. Such incidents are defined as extreme behavior, either outstandingly effective or ineffective with respect to attaining the general aims of the activity. The procedure has considerable efficiency because of the use of only the extremes of behavior. It is well known that extreme incidents can be more accurately identified than behavior which is more nearly average in character. One of the primary aims of scientific techniques is to insure objectivity for the observations being made and reported. Such agreement by independent observers can only be attained if they are all following the same set of rules. It is essential that these rules be clear and specific. In most situations the following specifications will need to be established and made explicit prior to collecting the data: a. The situations observed. The first necessary specification is a delimitation of the situa- tions to be observed. This specification must include information about the place, the persons, the conditions, and the activities. Such specifications are rather easily defined in many instances. For example, such brief specifications as observations of "the behavior in class- rooms of regularly employed teachers in a specified high school while instructing students during class periods," constitute a fairly adequate definition of a situation of this type. In complex situations it is probably essential not only that the specifications with respect to the situation be relatively complete and specific, but also that practical examples be provided to assist the observer in deciding in an objective fashion whether or not a specific behavior should be observed and recorded. b. Relevance to the general aim. After the decision has been made that a particular situa- tion is an appropriate one for making observations, the next step is to decide whether or nota specific behavior which is observed is relevant to the general aim of the activity as defined in the section above. For example, if the general aim of the activity was defined as sustained high quality and quantity of production, it might be difficult to decide whether ornot to include an action such as encouraging an unusually effective subordinate to get training that would assist him in developing his ability in an avocational or recreational activity not related to his work. In this case, it might be specified that any action which eitherdirectly or indirectly could be expected over a long period of time to have a significant effecton the general aim should be included. If it could not be predicted with some confidencewhether this effect would be good or bad, it should probably not be considered. The extent of detail required to obtain objectivity with respect to this type of decision depends to a considerable degree on the background and experiences of the observers with respect to this activity. For example, supervisors with substantial experience in a particular company can be expected to agree on whether or not a particular behavior is relevant to the attainment of the general aim. On the other hand, if outside observers were to be used, it would probably be necessary to specify in considerable detail the activities that can be expected to have an effect on the general aim. c . Extent of effect on the general aim. The remaining decision that the observer must makeis how important an effect the observed incident has on the general aim. It is necessary tospecify two points on the scale of importance: (a) a level of positive contributions to the gen- eral aim in specific terms, preferably including a concrete example, and (b) the correspondinglevel of negative effect on the general aim expressed in similar terms. A definition which has been found useful is that an incident is critical if it makes a "sig- nificant" contribution, either positively or negatively, to the general aim of the activity. The definition of "significant" will depend on the nature of the activity. If the general aim of the activity is in terms of production, a significant contribution might be one which caused, or might have caused, an appreciable change in the daily production of the department either in the form of an increase or a decrease. In certain specific situations, it might be desirable and possible to set up a quantitative criterion such as saving or wasting 15 minutes of an average worker's production. In some situations, a definition of significance might be set up in terms of dollars saved or lost both directly and indirectly. Actions which influence the attitudes of others are more difficult to evaluate objectively. Perhaps the best we might be able to do is to state it in terms of a probability estimate. For ably close for most of the items. Items on such matters as meeting production requirements and accepting changes in jobs are higher in terms of the recorded than the recalled incidents. The fact that items such as wasting time and assisting on problems are lower for the recalled incidents suggests that part of this discrepancy lies in the interpretations of the category definitions. The classifying of recorded incidents was done by the foremen, while the classification of the recalled incidents was done by the research workers. In fairness, it should also be noted that the definitions used by the research workers were rewritten before they were incorporated in the foremen's manuals. On the whole, it seems reasonable to assume that, if suitable precautions are taken, recalled incidents can be relied on to provide adequate data for a fairly satisfactory first approximation to a statement of the requirements of the activity. Direct observations are to be preferred, but the efficiency, immediacy, and minimum demands on cooperating personnel which are achieved by using recalled incident data frequently make their use the more practical procedure. Another practical problem in collecting the data for describing an activity refers to the problem of how it should be obtained from the observers. This applies especially to the problem of collecting recalled data in the form of critical incidents. Four procedures have been used and will be discussed briefly below: a. Interviews. The use of trained personnel to explain to observers precisely what data are desired and to record the incidents, making sure that all necessary details are supplied, is probably the most satisfactory data collection procedure. This type of interview is somewhat different from other sorts of interviews and a brief summary of the principal factors involved will be given. (i) Sponsorship of the study. If a stranger to the observers is collecting the data, it is ordi- narily desirable to indicate on what authority the interview is being held. This part should be as brief as possible to avoid any use of time for a prolonged discussion of a topic irrelevant to the purpose of the interview. In many instances all that needs to be said is that someone known and respected by the observer has suggested the interview. (ii) Purpose of the study. This should also be brief and ordinarily would merely involve a statement that a study was being made to describe the requirements of the activity. This would usually be cast in some such informal form as, "We wish to find out what makes a good citizen," or, "We are trying to learn in detail just what successful work as a nurse includes." In cases where there is some hesitation about cooperating or a little more explanation seems desirable, a statement can be added concerning the value and probable uses of the results. This frequently takes the form of improving selection and training procedures. In some instances, it would involve improving the results of the activity. For example, the interviewer might say, "In order to get better sales clerks we need to know just what they do that makes them especially effective or ineffective," or, "If parents are to be more effective, we need to be able to tell them the things they do that are effective and ineffective." (iii) The group being interviewed. If there is any likelihood of a person feeling, "But, why ask me?" it is desirable to forestall this by pointing out that he is a member of a group which is in an unusually good position to observe and report on this activity. The special qualifications of members of this group as observers can be mentioned briefly, as, "Supervisors such as yourself are constantly observing and evaluating the work of switchboard girls," or, "Students are in an unusually good position to observe the effectiveness of their teachers in a number of ways." (iv) The anonymity of the data. Especially for the collection of information about inef- fective behavior, one of the principal problems is to convince the observer that his report cannot harm the person reported on in any way. Usually he also needs to be convinced that the person reported on will never know that he has reported the incident. Assurances are not nearly so effective in this situation as actual descriptions of techniques to be used in handling the data, which enable the observer to judge for himself how well the anonymity of the data will be guarded. Under no circumstances should the confidences of the reportees be violated in any way. The use of sealed envelopes, avoidance of identifying information, the mailing of data immediately to a distant point for analysis, and similar techniques are helpful in establishing the good faith of the interviewer in taking all possible precautions to safeguard the incidents reported. (v) The question. The most crucial aspect of the data collection procedure is the questions asked the observers. Many studies have shown that a slight change in wording may produce a substantial change in the incidents reported. For example, in one study the last part of one of the specific questions asked was, "Tell just how this employee behaved which caused a noticeable decrease in production." This question resulted in almost all incidents reported having to do with personality and attitude behaviors. This part of the question was changed to, "Tell just what this employee did which caused a noticeable decrease in production." This second question produced a much broader range of incidents. To the question writer "how he behaved" and "what he did" seemed like about the same thing. To the foremen who were reporting incidents "how he behaved" sounded as if personality and attitudes were being studied. The subtle biases involved in the wording of questions are not always so easily found. Questions should always be tried out with a small group of typical observers before being put into general use in a study. The question should usually refer briefly to the general aim of the activity. This aim might be discussed more fully in a preliminary sentence. It should usually state that an incident, actual behavior, or what the person did is desired. It should briefly specify the type of behavior which is relevant and the level of importance which it must reach to be reported. It should also tie down the selection of the incidents to be reported by the observer in some way, such as asking for the most recent observation, in order to prevent the giving of only the more dramatic or vivid incidents, or some other selected group, such as those which fit the observer's stereotypes. An effective procedure for insuring that the interpretation of the persons being interviewed is close to that intended is to request a sample of persons typical of those to be interviewed to state in their own words what they understand they have been asked to do. These persons should be selected so as to represent all types who will be interviewed. From a study of their interpretations, necessary revisions can be made to insure that all interviewees will be in agreement as to the nature of the incidents they are to provide. (vi) The conversation. The interviewer should avoid asking leading questions after the main question has been stated. His remarks should be neutral and permissive and should show that he accepts the observer as the expert. By indicating that he understands what is being said and permitting the observer to do most of the talking, the interviewer can usually get unbiased incidents. If the question does not seem to be understood, it can be repeated with some reference to clarifying just what is meant by it. If the observer has given what seems like only part of the story, he should be encouraged by restating the essence of his remarks. This usually tends to encourage him to continue and may result in his bringing out many relevant details that the interviewer did not know the situation well enough to ask for. In some cases, it is desirable to have the interviews recorded electrically and transcribed. This increases the work load substantially, and trained interviewers can usually get satisfactory reports at the time or by editing their notes shortly after the interview. Usually the interviewer should apply certain criteria to the incidents while they are being collected. Some of the more important criteria are: (a) is the actual behavior reported; (b) was it observed by the reporter; (c) were all relevant factors in the situation given; (d) has the observer made a definite judgment regarding the criticalness of the behavior; (e) has the observer made it clear just why he believes the behavior was critical. In Fig. 3 is shown a sample of the type of form used by interviewers to collect critical in- cidents. Of course the form must be adapted to the needs of the specific situation. b. Group interviews. Because of the cost in time and personnel of the individual interview, a group interview technique has been developed. This retains the advantages of the individual interview in regard to the personal contact, explanation, and availability of the interviewer to answer questions. To some extent it also provides for a check on the data supplied by the interviewees. Its other advantages are that the language of the actual observer is precisely reproduced and the time for editing the interviews is virtually eliminated. The method consists of having the interviewer give his introductory remarks to a group very much as he would do in an individual interview. There is an opportunity for questions and clarification. Then each person is asked to write incidents in answer to specific questions contained on a specially prepared form. The size of the group which can be handled effectively will vary with the situation. If the group is fairly small, it is usually possible for the interviewer to read the responses of each member of the group to the first question and make sure that he understands what is wanted. There seems to be a certain amount of social facilitation, and the results in most situations have been excellent. In the report of the first use of this procedure by Wagner (65), the amount of interviewer time required per usable incident was 4.3 minutes for the group interview procedure as compared with 15.7 minutes for individual interviews. The quality of these incidents, obtained from officers in the United States Air Force, appeared to be about the same for the two situations. others for review. Although there is no guarantee that results agreed on by several workers will be more useful than those obtained from a single worker, the confirmation of judgments by a number of persons is usually reassuring. The usual procedure is to sort a relatively small sample of incidents into piles that are related to the frame of reference selected. After these tentative categories have been established, brief definitions of them are made, and additional incidents are classified into them. During this process, needs for redefinition and for the de- velopment of new categories are noted. The tentative categories are modified as indicated and the process continued until all the incidents have been classified. The larger categories are subdivided into smaller groups and the incidents that describe very nearly the same type of behavior are placed together. The definitions for all the categories and major headings should then be re-examined in terms of the actual incidents classified under each. c. General behaviors. The last step is to determine the most appropriate level of specific- ity-generality to use in reporting the data. This is the problem of weighing the advantages of the specificity achieved in specific incidents against the simplicity of a relatively small number of headings. The level chosen might be only a dozen very general behaviors or it might be several hundred rather specific behaviors. Practical considerations in the immediate situation usually determine the optimal level of generality to be used. Several considerations should be kept in mind in establishing headings for major areas and in stating critical requirements at the selected level of generality. These are listed below: (i) The headings and requirements should indicate a clear-cut and logical organization. They should have a discernible and easily remembered structure. (ii) The titles should convey meanings in themselves without the necessity of detailed definition, explanation, or differentiation. This does not mean that they should not be defined and explained. It does mean that these titles, without the detailed explanation, should still be meaningful to the reader. (iii) The list of statements should be homogeneous; i.e., the headings for either areas or requirements should be parallel in content and structure. Headings for major areas should be neutral, not defining either unsatisfactory or outstanding behaviors. Critical requirements should ordinarily be stated in positive terms. (iv) The headings of a given type should all be of the same general magnitude or level of importance. Known biases in the data causing one area or one requirement to have a dispro- portionate number of incidents should not be reflected in the headings. (v) The headings used for classification and reporting of the data should be such that find- ings in terms of them will be easily applied and maximally useful. (vi) The list of headings should be comprehensive and cover all incidents having signifi- cant frequencies. 5. Interpreting and Reporting It is never possible in practice to obtain an ideal solution for each of the practical problems involved in obtaining a functional description of an activity. Therefore, the statement of requirements as obtained needs interpretation if it is to be used properly. In many cases, the real errors are made not in the collection and analysis of the data but in the failure to interpret them properly. Each of the four preceding steps, (a) the determination of the general aim, (b) the specification of observers, groups to be observed, and observations to be made, (c) the data collection, and (d) the data analysis, must be studied to see what biases have been introduced by the procedures adopted. If there is a division of opinion as to the general aim and one of the competing aims is selected, this should be made very clear in the report. If the groups on whom the observations are made are not representative of the relevant groups involved, they must be described as precisely as possible. The aim of the study is usually not a functional de- scription of the activity as carried on by this sample but rather a statement relating to all groups of this type. In order to avoid faulty inferences and generalizations, the limitations imposed by the group must be brought into clear focus. Similarly, the nature of judgments made in collecting and analyzing the data must be carefully reviewed. While the limitations need to be clearly reported, the value of the results should also be emphasized. Too often the research worker shirks his responsibility for rendering a judgment concerning the degree of credibility which should be attached to his findings. It is a difficult task, but if the results are to be used, someone will have to make such a judgment, and the original investigator is best prepared to make the necessary evaluations either for the general case or for certain typical specific examples. USES OF THE CRITICAL INCIDENT TECHNIQUE The variety of situations in which the collection of critical incidents will prove of value has only been partially explored. In the approximately eight years since the writer and his colleagues began a systematic formulation of principles and procedures to be followed in collecting this type of data, a fairly large number of applications has been made. The applications will be discussed under the following nine headings: (a) Meas- ures of typical performance (criteria); (b) measures of proficiency (standard samples); (c) training; (d) selection and classification; (e) job design and purification; (f) operating procedures; (g) equipment design; (h) motivation and leadership (attitudes); (i) coun- seling and psychotherapy. Space is not available here to describe these various applications in detail. However, a brief description of the types of application that have been made, along with brief illustrative examples and references, will be presented. Some of the studies involve several of the types of applications to be discussed. The presentation is not intended to be complete, but rather to give the reader interested in further study some orientation and guidance. Measures of typical performance (criteria). The simplest and most natural application of a systematically collected set of critical incidents is in terms of the preparation of a statement of critical requirements and a check list or some similar type of procedure for evaluating the typical performance of persons engaged in this activity. If an observational check list that includes all of the important behaviors for the activity is available, the performance of the individual can be objectively evaluated and recorded by merely making a single tally mark for each observation. Such records provide the essential basis for criterion data which are sufficiently detailed and specific for special purposes but at the same time can be combined into a single over-all evaluation when this is desirable. Such a procedure was first suggested and tried out in connection with developmental studies of the American Institute for Research. These in- cluded: Preston's study of officers for the United States Air Force (52); Nagay's study on air route traffic controllers for the Civil Aeronautics Administration (49); and M. H. Weislogel's study on research personnel for the Office of Naval Research (69). Another American Institute for Research study was reported by R. B. Miller and the present author (21). This was a performance record form for hourly wage employees developed in cooperation with personnel of the Delco-Remy Division of the General Motors Corporation, the Employment Practices Division of that corporation, and the Industrial Relations Center of the University of Chicago. The same authors have developed similar performance records for salaried employees, and foremen and supervisors (22, 23). The principles and procedures underlying this type of evaluation of performance have been published elsewhere (14, 15, 17). A number of important contributions to the development of functional descriptions and standards of performance have been made by other groups using the critical incident technique. One of the most notable of these is the development by Hobbs et al. (3, 31), of Ethical Standards of Psychologists. More than 1,000 critical incidents involving ethical problems of psychologists were contributed by the members of the American Psychological Association. It is believed that this represents the first attempt to use empirical methods to establish ethical standards. Because of the importance of this study, and the generality of some of the problems involved, certain of the conclusions reported by the Committee on Ethical Standards for Pyschology in their introductory statement will be quoted here. First, it is clear that psychologists believe that ethics are important; over two thousand psychologists were sufficiently concerned with the ethical obligations of their profession to contribute substantially to the formulation of these ethical standards. Second, psychologists believe that the ethics of a profession cannot be prescribed by a committee; ethical standards must emerge from the day-by-day value commitments made by psychologists in the practice of their profession. Third, psychologists share a conviction that the problems of men, even those involving values, can be studied objectively; this document summarizes the results of an effort to apply some of the tech- niques of social science to the study of ethical behavior of psychologists. Fourth, psychologists are aware that a good code of ethics must be more than a description of the current status of ethics in the profession; a code must embody the ethical aspirations of psychologists and encourage changes in behavior, bringing performance ever closer to aspiration. Fifth, psychologists appreciate that process is often more important than product in influencing human behavior; the four years of widely-shared work in developing this code are counted on to be more influential in changing ethical practices of psychologists than will be the publication of this product of their work. Finally, psychologists recognize that the process of studying ethical standards must be a continuing one; occasional publications such as this statement mark no point of conclusion in the ongoing process of defining ethical standards—they are a means of sharing the more essential discipline of examining professional experience, forming hypotheses about professional conduct, and testing these hypotheses by reference to the welfare of the people affected by them (3, p. v). In addition to the study by Smit mentioned in a previous section (58), several other studies on the use of the critical incident procedures as a basis for evaluating teaching effectiveness have been reported. One of these was a study conducted under the joint sponsorship of the Educational Research Corporation and the Harvard University Graduate School of Education with funds provided by the New England School Develop- ment Council and the George F. Milton Fund. This was an exploratory study of teacher competence reported by Domas (6). Approximately 1,000 critical incidents were collected from teachers, principals, and other supervisors. Although this was an exploratory study, it was felt that it made an important contribution to the general problem of relating salary to teacher competence. The second of these studies was conducted as part of the teacher characteristics study sponsored by the American Council on Education and subsidized by the Grant Founda- tion. This study is reported by Jensen (32). Teachers, administrators, and teachers in training in the Los Angeles area contributed more than 1,500 critical incidents of teacher behavior. The incidents were classified under personal, professional, and social qualities. The category formulation indicated that there were about 20 distinct critical requirements. (61) for the Human Resources Research Center. This study was based on critical incidents collected in previous studies of the American Institute for Research and reported the development and preliminary evaluation of the Officer Situations Test. This test was designed to measure nonintellectual aspects of officer performance through the use of 16 situational problems requiring about six hours of testing time. Because of the great difficulty in obtaining valid and reliable measures of typical performance, accurate measures of proficiency are essential for many types of activities. It is apparent that a comprehensive set of critical incidents can be of great value in constructing such measures. Training. Many of the applications of the critical incident technique to training problems have been carried out for the military in special situations so that the reports are classified security information. In addition to work by Preston, Glaser, and R. L. Weislogel, R. B. Miller and Folley have utilized critical incidents in establishing training requirements for specific types of maintenance mechanics (47) in a study for the Human Resources Research Center. Similarly, Ronan has used critical incidents as a basis for developing a program of training for emergency procedures in multi-engine aircraft (54) in a study for the United States Air Force Human Factors Operations Research Laboratory. On the basis of several thousand incidents reported by aircrew personnel regarding emergencies, three evaluation devices were prepared. These involved a conventional type multiple-choice test; a special multiple-choice test designed to measure the individual's information concerning the important cues in the emergency situation, the appropriate actions to be taken, and the basic troubles or causes of the emergency; and a "flight check" to be used in evaluating the performance of aircrew members in electronic flight simulators. The obvious relevance of the behaviors involved in critical incidents and the specific details included make such incidents an ideal basis for developing training programs and training materials. A recent study by Collins (5) uses critical incidents as a basis for evaluating the effectiveness of a training program. The types of incidents reported by mothers after a two-week training course were significantly different from those reported at the beginning of the program in a number of aspects relevant to the objectives of the program. The critical incidents appeared to provide a much more sensitive basis for revealing changes than other procedures used. Selection and classification. Until recently, the customary approach of the research psychologist to the development of tests for selection and classification purposes has been as follows: A very brief period was given to study of the job. Following this, a wide variety of selection procedures was administered to a group of applicants or employees, and follow-up data were gathered. Since the research psychologist had little confidence in the accuracy of his analysis of the psychological elements required by the job, there was a tendency to try everything that was available and seemed even remotely related to the tasks involved. This has been called the "shotgun approach." It was hoped that with a wide scatter at least a few of the tests would pay off. The critical incident technique has lent substantial support to the more thorough study of the job prior to initiating testing procedures. There is increasing feeling at the present time that a much larger percentage of the investigator's time should be spent on determining the critical requirements of the job, so that the psychologist will have sufficient confidence in his tentative conclusions as to the nature of the important selection procedures to permit their use on a tentative basis prior to the collection of empirical follow-up data. This is especially important in those situations where the follow-up requires a very long period of time or where the number of cases that can be followed up is so small that definitive findings cannot be anticipated. One of the most important requirements for developing a system of job analysis that will facilitate a relatively accurate identification of the important job elements for a specific task is to establish a clear and specific set of definitions for these job elements in behavioral terms. The American Institute for Research has carried out a series of projects on this problem. The first of these was a study undertaken by Wagner under the sponsorship of the United States Air Force School of Aviation Medicine to define the requirements of aircrew jobs in terms of specific job elements (67, 68). Several thousand critical incidents were gathered from aircrew members, and these were classified into 24 job elements. These job elements were inductively formulated from the critical incidents and were grouped under the four area headings: (a) learning and thinking; (b) observation and visualization; sensory-motor coordination; and motives, temperament, and lead- ership. The development of more than 100 proficiency tests to measure each of the various critical behaviors included in the 24 tentatively proposed job elements was reported by Hahn (30) for the School of Aviation Medicine. These tests were administered to a group of approximately 500 high school senior boys, and the intercor-relations were used to reformulate the tentative job elements. In a study just completed by Taylor (62) for the Human Resources Research Center, the results of applying an analytical procedure developed by Horst to study the interrelationships involved are reported. This analysis led to the formulation of a new set of 20 job elements for each of which a selection test has been developed. These tests have been administered to several hundred aviation cadets and follow-up data on their success in aircrew training should be available soon. A similar project based on critical incidents collected from various civilian jobs has been reported by the present author (2, 18, 19). The Flanagan Aptitude Classification Test Series, published in 1953, provides aptitude measures for 14 critical job elements. The Applicant Inventory, also published in 1953, measures attitudes predictive of job adjustment for hourly wage employees. An effort to adapt the critical incident technique to the problem of developing civil service examinations is reported by Wager and Sharon (64). In an exploratory study they collected about 100 incidents regarding on-the-job behaviors of maintenance technicians. These incidents were used as a basis for determining job requirements in terms of be- havior, and test items were developed for use in selecting applicants who could be expected to meet these requirements. Another study that used critical incidents as a basis for developing tests to predict performance was carried out by O'Donnell (51). His test, designed to predict success in dentistry, was based on critical incidents collected by Wagner. The test includes items designed to predict, in part, the following three general areas: (a) demonstrating technical proficiency; (b) handling patient relationships; and (c) accepting professional responsibility. A follow-up study indicated moderate validity for these materials. One of the few studies known to the author in which the critical incident technique was used in a project carried on outside the United States is Emons' doctor's dissertation (8). This study, carried out at the University of Liege, investigated the aptitudes of effective sales personnel in a large department store. A group of 40 supervisors provided 228 critical incidents. Nine categories were formulated from this group of incidents and recommendations made for an aptitude test to improve current selection procedures. 5. Job design and purification. Inadequate attention has been given to the scientific design of jobs to promote over-all efficiency. Where a team has several different types of tasks to perform, it is frequently possible to design each of the team member's jobs so that only a few of the several tasks are involved. If the jobs have been studied by use of the critical incident technique, it may be possible to select and train each team member for only two or three of the critical job elements. This tends to maximize the effectiveness of performance with respect to each of the various types of tasks. Although such procedures have nearly always been informally used in planning the work of teams, the critical incident technique facilitates the collection of the data essential to this type of job purification. Some preliminary work on this problem has been carried out at the American Institute for Research. Recommendations resulting from these studies for reducing the number of job elements required in certain common maintenance jobs are expected to lead to a saving of millions of dollars in training costs as well as to improving the effectiveness of job performance. Operating procedures. Another application of critical incidents which has not been adequately exploited is the study of operating procedures. Detailed factual data on successes and failures that can be systematically analyzed are of great importance in improving the effectiveness and efficiency of operations. Such information can be efficiently collected by means of the critical incident technique. Examples of such studies are provided by a series of three projects carried out by the American Institute for Research under the sponsorship of the United States Air Force School of Aviation Medicine. The first of these involves the collection of critical incidents relating to near accidents in flying reported by Vasilas, Fitzpatrick, DuBois, and Youtz (63). More than 1,700 critical incidents were collected from pilots and other aircrew members by procedures developed for this study. These incidents pointed to possible improvements in training job design and equipment design as well as in operat- ing procedures. The second of these studies was specifically concerned with the effect of the age of pilots and other crew members on aircrew operations. This study was reported by Shriver (56), and included tentative suggestions regarding various modifications in operating procedures. The third study in this series, reported by Goodenough and Suttell (26), involved the collection of critical incidents regarding the impairment of human efficiency in emer- gency operations. These incidents provide a detailed statement of both the types of stresses that impair performance and the types of performance that are impaired under specific conditions. More than 2,000 critical incidents were collected in which impairment in performance on operational assignments was observed. These incidents were collected in Alaska and the Far East as well as in operational commands in the United States. This report contains suggestions for improving operations in emergency situations. Equipment design. An application closely related to that just discussed involves the collection of critical incidents to improve the design of equipment. Reports of specific in- such as establishing standards, determining requirements, or evaluating results should have wide applicability. The applications of the critical incident technique which have been made to date are discussed under the following nine headings: (a) Measures of typical performance (criteria); (b) measures of proficiency (standard samples); (c) training; (d) selection and classification; (e) job design and purification; (/) operating procedures ; (g) equipment design; (h) motivation and leadership (attitudes); (i) counseling and psychotherapy. In summary, the critical incident technique, rather than collecting opinions, hunches, and estimates, obtains a record of specific behaviors from those in the best position to make the necessary observations and evaluations. The collection and tabulation of these observations make it possible to formulate the critical requirements of an activity. A list of critical behaviors provides a sound basis for making inferences as to requirements in terms of aptitudes, training, and other characteristics. It is believed that progress has been made in the development of procedures for determining activity requirements with objectivity and precision in terms of well-defined and general psychological categories. Much remains to be done. It is hoped that the critical incident technique and related developments will provide a stable foundation for procedures in many areas of psychology. REFERENCES 1. ALLEN, CD. Critical requirements in interpersonal behavior. Unpublished senior thesis, Princeton Univer., 1950. 2. AMERICAN INSTITUTE FOR RESEARCH. The applicant inventory. Pittsburgh: American Institute for Research, 1954. 3. AMERICAN PSYCHOLOGICAL ASSOCIATION. COMMITTEE ON ETHICAL STANDARDS FOR PSYCHOLOGY. Ethical standards of psychologists. Washington, D. C: Amer. Psychol. Ass., 1953. 4. BARNHART, R. E. The critical requirements for school board membership based upon an analysis of critical incidents. Unpublished doctor's dissertation, Indiana Univer., 1952. 5. COLLINS, MARJORIE G. Selected methods applied to the evaluation of a parent training course in child management. Unpublished doctor's dissertation, Univer. of Pittsburgh, 1954. 6. DOMAS, S. J. Report of an exploratory study of teacher competence. Cambridge: New England School Development Council, 1950. 7. EILBERT, L. R. A study of emotional immaturity utilizing the critical incident technique. Univer. Pittsburgh Bull., 1953, 49, 199-204. 8. EMONS, V. L'analyse de la fonction de vendeuse de grand magasin par la methode des exigencies critiques. Unpublished doctor's dissertation, Univer. of Liege, Belgium, 1952. 9. ERICKSEN, S. C. Development of an objective proficiency check for private pilot certification. Washington: Civil Aeronautics Administration, 1951. (Div. of Res., Rep. No. 95.) 10. ERICKSEN, S. C. Development of a light plane proficiency check to predict military flying success. USAF, Hum. Re-sour. Res. Cent. Tech. Rep., 1952, No. 52-6. 11. FINKLE, R. B. A study of the critical requirements of foremanship. Univer. Pittsburgh Bull., 1950,46,291-297. (Abstract) 12. FITTS, P. M., & JONES, R. E. Psychological aspects of instrument display. 1. Analysis of 270 "pilot error" experiences in reading and interpreting aircraft instruments. Dayton, O.: U. S. Air Force, Air Materiel Command, Wright-Patterson Air Force Base, 1947. (Mem. Rep. TSEAA-694-12A.) 13. FLANAGAN, J. C. The aviation psychology program in the Army Air Forces. Washington: U. S. Government Printing Office, 1947. (AAF Aviat. Psychol. Program Res. Rep. No. 1.) 14. FLANAGAN, J. C. A new approach to evaluating personnel. Personnel, 1949, 26, 35-42. 15. FLANAGAN, J. C. Critical requirements: a new approach to employee evaluation. Personnel Psychol., 1949, 2, 419-425. 16. FLANAGAN, J. C. Use of comprehensive rationales. Educ. psychol. Measmt, 1951, 11, 151- 155. 17. FLANAGAN, J. C. Principles and procedures in evaluating performance. Personnel, 1952, 28, 373-386. 18. FLANAGAN, J. C. Flanagan aptitude classification tests. Chicago: Science Research Associates, 1953. 19. FLANAGAN, J. C. Improving personnel selection. Publ. Personnel Rev., 1953, 14, 107-112. 20. FLANAGAN, J. C, et al. Critical requirements for research personnel. Pittsburgh : American Institute for Research, 1949. 21. FLANAGAN, J. C, MILLER, R. B., BURNS, R. K., HENDRIX, A. A., STEWART, B., PRESTON, H. O., & WEST, E. D. The performance record for hourly employes. Chicago: Science Research Associates, 1953. 22. FLANAGAN, J. C, MILLER, R. B., BURNS, R. K., HENDRIX, A. A., STEWART, B., PRESTON, H. O., & WEST, E. D. The performance record for non-supervisory salaried employes. Chicago: Science Research Associates, 1953. 23. FLANAGAN, J. C, MILLER, R. B., BURNS, R. K., HENDRIX, A. A., STEWART, B., PRESTON, H. O., & WEST, E. D. The performance record for foremen and supervisors. Chicago: Science Research Associates, 1953. 24. FOLLEY, J. D., JR. Development of a list of critical requirements for retail sales personnel from the standpoint of customer satisfaction. Unpublished master's thesis, Univer. of Pittsburgh, 1953. 25. GOLDFARB, A. Use of the critical incident technique to establish areas of change accompanying psychotherapy: II. Relationship to diagnostic group. Unpublished master's thesis, Univer. of Pittsburgh, 1952. 26. GOODENOUGH, D. R., & SUTTELL, BARBARA J. The nature and extent of impair- ment of human efficiency in emergency operations: the field study. Pittsburgh: American Institute for Research, 1952. 27. GORDON, T. The airline pilot: a survey of the critical requirements of his job and of pilot evaluation and selection procedures. Washington: Civil Aeronautics Administration, 1947. {Div. of Res., Rep. No. 73.) 28. GORDON, T. The development of a standard flight-check for the airline transport rating based on the critical requirements of the airline pilot's job. Washington: Civil Aeronautics Administration, 1949. (Div. of Res., Rep. No. 85.) 29. GORDON, T. The development of a method of evaluating flying skill. Personnel PsychoL, 1950, 3, 71-84. 30. HAHN, C. P. Measurement of individual differences with respect to critical job requirements. USAF Sell. Aviat. Med., 1954, Proj. No. 21-29-014, No. 2. 31. HOBBS, N. The development of a code of ethical standards for psychology. Amer. Psychologist, 1948, 3, 80-84. 32. JENSEN, A. C. Determining critical requirements for teachers. /. exp. Educ, 1951, 20, 79-86. 33. KONIGSBURG, D. The development and preliminary evaluation of an instructor check list based on the critical incident technique. Unpublished doctor's dis- sertation, Univer. of Pittsburgh, 1954. 34. KRUMM, R. L. Critical requirements of pilot instructors. USAF, Hum. Resour. Res. Cent., Tech. Rep., 1952, No. 52-1. 35. KRUMM, R. L. Development of a measure of pilot instructor proficiency based on the critical requirements of the instructor's job. Pittsburgh: American Institute for Research, 1953. 36. MARLEY, F. W. Individual differences in critical aircrew elements. I. The deter- mination of critical proficiency requirements for B-29 combat crews. USAF Sch. Aviat. Med., 1952, Proj. No. 21-29-014, No. 1. 37. MARLEY, F. W. The development of performance flight checks for B-29 combat crews. USA F, Hum. Resour. Res. Lab., 1952, Rep. No. 19. 38. MELLETT, T. P. Use of the critical incident technique to establish areas of change accompanying psychotherapy: III. Differences among therapists. Unpublished master's thesis, Univer. of Pittsburgh, 1952. 39. MILLER, G. S. Development of B-36 bombardment objective flight checks and proficiency measures for use with reconnaissance crews. Pittsburgh: American Institute for Research, 1953. 40. MILLER, N. E. Psychological research on pilot training. Washington: U. S. Gov- ernment Printing Office, 1947. (AAF Aviat. Psychol. Program. Res. Rep. No. 8.) 41. MILLER, R. B. Anticipating tomorrow's maintenance job. USA F, Hum. Resour. Res. Cent., Res. Rev., 1953, No. 53-1. 42. MILLER, R. B. A method for determining human engineering design requirements for training equipment. Dayton, O.: U. S. Air Force, Air Development Center, Wright-Patterson Air Force Base, 1953. {Tech. Rep. 53-135.) 43. MILLER, R. B. Handbook on training and training equipment design. Dayton, O.: U. S. Air Force, Air Development Center, Wright-Patterson Air Force Base, 1953. {Tech. Rep. 53-136.) 44. MILLER, R. B. A method for man-machine task analysis. Dayton, O.: U. S. Air Force, Air Development Center, Wright-Patterson Air Force Base, 1953. {Tech. Rep. 53-137.) 45. MILLER, R. B. Human engineering design schedule for training equipment. Dayton, O.: U. S. Air Force, Air Development Center, Wright-Patterson Air Force Base, 1953. {Tech. Rep. 53-138.) 46. MILLER, R. B., & FLANAGAN, J. C. The performance record: an objective merit- rating procedure for industry. Amer. Psychologist, 1950, 5, 331-332. (Abstract) 47. MILLER, R. B., & FOLLEY, J. D., JR. The validity of maintenance job analysis from the prototype of an electronic equipment. Pittsburgh: American Institute for Research, 1952. 48. NAGAY, J. A. The development of a procedure for evaluating the proficiency of air route traffic controllers. Washington: Civil Aeronautics Administration, 1949. {Div. of Res., Rep. No. 83.) 49. NAGAY, J. A. The airline tryout of the standard flight-check for the airline transport rating. Washington: Civil Aeronautics Administration, 1949. {Div. of Res., Rep. No. 88.) 50. NEVINS, CHARLOTTE I. An analysis of reasons for the success or failure of bookkeepers in sales companies. Unpublished master's thesis, Univer. of Pittsburgh, 1949. 51. O'DONNELL, R. J. The development and evaluation of a test for predicting dental student performance. Univer. Pittsburgh Bull., 1953, 49, 240-245. (Abstract) 52. PRESTON, H. O. The development of a procedure for evaluating officers in the United States Air Force. Pittsburgh: American Institute for Research, 1948. 53. PRESTON, H. O. Events affecting reenlistment decisions. Pittsburgh: American Institute for Research, 1953. 54. RONAN, W. W. Training for emergency procedures in multi-engine aircraft. Pittsburgh: American Institute for Research, 1953. 55. RUCH, F. L. Incidents of leadership in combat. USAF, Hum. Resour. Res. Inst., Res. Memo, 1953, No. 3. 56. SHRIVER, BEATRICE M. Age and behavior: a study of the effects of aging on aircrew performance. USAF Sch. Aviat. Med., 1953, Proj. No. 21-0202-0005, No. 3. 57. SIVY, J., & LANGE, C. J. Development of an objective form of the Leaders Reaction Test. AGO, Personnel Res. Branch, PRS Rep. 1952, No. 930. 58. SMIT, JO ANNE. A study of the critical re-requirements for instructors of general psychology courses. Univer. Pittsburgh Bull., 1952, 48, 279-284. (Abstract) 59. SMITH, R. G., JR., & STAUDOHAR, F. T. Technical requirements of basic training tactical instructors. USAF, Hum. Resour. Res. Cent., Tech. Rep., 1954, No. 54-7.