IN THE MATTER OF Docket No. 01-20-O
Inter-Tribal Council, Inc., Indian Vocational Education Program Proceeding
Jeannie Gleaves, Director, Indian Vocational Program, Miami, Oklahoma, for the Inter-Tribal Council, Inc.
Daphna Krim, Esq., of the Office of the General Counsel, United States Department of Education, Washington, D.C., for the Assistant Secretary for Vocational and Adult Education.
The Inter-Tribal Council, Inc. (Council) seeks a review of a decision by the Secretary not to include Council among the 30 applicants selected in a competitive grant process for the award of a grant under the Native American Vocational and Technical Education Program. 66 Fed Reg. 560 (2001). Council asserts that the procedure employed by the Department to evaluate its application was flawed and that various errors were made by the reviewers that warranted an increase in its overall score. Based on the analysis, infra, the final rating of Reviewer #2 is adjusted from 79 to 80 points. No adjustment to the final ratings assigned by Reviewer #1, #3, and #4 is warranted.
The Native American Vocational and Technical Education Program provides grants to improve vocational and technical education programs that are consistent with the purposes of the Carl D. Perkins Vocational and Applied Technology Education Act. Pub. L. No. 88-210, 77 Stat. 403 (1963), as amended by the Carl D. Perkins Vocational and Applied Technology Education Amendments of 1998, Pub. L. No. 105-332, 112 Stat. 3076 (1998) (to be codified at 20 U.S.C. § 2301 et seq.). In awarding grants and entering into contracts, the Secretary shall ensure that the grants and contracts will improve vocational and technical education programs. 20 U.S.C. § 2326(e). Pursuant to this mandate, the Secretary published on January 3, 2001, a notification that solicited applications for a limited number of grants. 66 Fed. Reg. 560 (2001). The Secretary received 73 applications and, presently, anticipates funding 30 applications based upon their standardized scores.
The Office of Vocational and Adult Education (ED) used eight panels, each comprised of four reviewers, to evaluate the applications. Each panel reviewed approximately nine applications. The applications were evaluated based upon eight categories of criteria, totaling 125 points, and two additional categories, totaling 15 points, and authorized under Section 116(e) of the Carl D. Perkins Vocational and Applied Technology Education Act. These categories and their criteria were set forth in the official notification in the Federal Register.
Council’s application was reviewed by a panel of four individuals and received raw scores of 63, 71, 73, and 79. These scores were standardized and then averaged. Council’s standardized score was 85.04 and ranked 42nd among the 73 applicants. This was 12 places removed from the last funded applicant that had a standardized score of 106.5. Council appeals its final score questioning the rating procedure and the rating scores assigned by the four reviewers.
A. RATING PROCEDURE
Council raises three concerns regarding the review process. First, it urges that all applications should be reviewed by one panel so that they will be rated from the same perspective. Second, it believes that the entire rating process of an application should occur on the same day, i.e. its reading, the discussion, and its final rating. It also urges that each panel member should rate the application separately and without any discussion with other panel members. Third, because of a lack of information, it questions how the standardization works to remove reviewer bias.
ED responds that it would take approximate eight weeks for one panel to review the 73 applications and that such an approach is unworkable. It adds that the usage of a statistical standardization process eliminates any problems of reviewer bias due to, for example, different professional perspectives. This mathematical formula permits scores determined by panelists to be compared fairly and yields an overall ranking of the applications relative to their merit.
As noted by ED, this issue was raised and addressed previously in In re Sisseton Wahpeton Community College, Dkt. No. 95-86-O, U.S. Dept. of Education (Aug. 7, 1995). The tribunal concluded there, as it does here, that the employment of the standardization program removes the reviewer bias from the assigned raw scores and, therefore, the usage of multiple panels is appropriate and fair for all applicants.
The written evaluative comments and the panel discussions are aspects of the process designed to promote a fair and reasonable evaluation of the applications through the production and exchange of analyses. Hence, a same-day evaluation, as urged by Council, is not necessary. Moreover, Council’s other suggestion -- a purely independent evaluation by each panelist – would preclude the fair exchange of ideas and information among the panelist and is inconsistent with the rating process. Therefore, it is rejected.
Lastly, ED provided an explanation of the standardization formula. It reflects an accepted statistical approach for ranking ordering evaluations of members of a group where more than one evaluator is employed. The method is used by the Department in other discretionary grant competitions.
B. OBJECTIONS TO SCORES ASSIGNED BY THE REVIEWERS
Initially, Council’s program is not a vocational and technical education program. Rather, it is an “overseer” type of program that provides counseling to individuals regarding the selection of a post secondary technical or vocational education, monitors their progress, supplies limited financial assistance, and provides assistance toward seeking employment. The broad areas of vocational education available to its target population are truck driving, customer services such as childcare and hospitality, health related services such as medical lab tech and medical assistant, auto/diesel technology such as auto repair technician, and service industries such as heating/air conditioning and industrial technology. The primary function of the tribunal at this stage of the award process is—
not to reevaluate the application and supplant the judgment of the reviewers with its judgment. The role of this tribunal is to ensure that the process of selecting successful grant recipients is fundamentally sound. Regarding the scoring aspect, it was held in Sisseton Wahpeton Community College at 6, that the score rendered by a reviewer with respect to each criterion should not be disturbed unless significant error exists. Such a standard recognizes that reasonable persons may differ in their evaluations and yet permits correction in the event of significant error.
In re Miccosukee Corporation, Dkt. No. 97-9-O, U.S. Dept. of Education (Mar. 24, 1997) at 5. The various concerns raised by Council will be considered below.
Part A: Needs for Project. In the needs category, the Secretary considers (1) the need for the proposed project based upon the magnitude or severity of the problem, (2) the magnitude of the need for the services or activities provided under the project as evidenced by, for example, data from surveys or tribal economic development plans, and (3) the extent to which the project will address specific gaps or weaknesses in services, infrastructure, or opportunities that exist.
The four reviewers rated the need for project factor at high end of inadequate (6 of 15 points by Reviewer #4) and mid adequate (8 points by Reviewers #1, #2, and #3). Council did not contest these scores.
Part B: Significance of the Project. In the significance category, the Secretary considers the effect of the proposed project toward improving the applicant’s program, local and other programs, and the likelihood of improvements in teaching and student achievement. The reviewers rated the significance of the project factor as low adequate (5 of 10 points by Reviewers #1 and #4), high adequate (7 points by Reviewer #2), and good (8 points by Reviewer #3).
Reviewer #2 noted five strengths and five weaknesses and Council disagrees with three of the weaknesses. The reviewer noted that the project proposes to build on the improvements in the educational program made under prior programs and cited, as a weakness, that a clarification of what had been done previously was not indicated. This information was important to establish the base level of the proposed project. Council responds that the program has been funded by the Department through four competitions and cited its statement under the Quality of Project Service category that it has had a successful completion rate over 80% and that 75% of these students became employed or went on to higher education. This response fails to address the reviewer’s concern and, therefore, the weakness is upheld.
Reviewer #2 noted that the project has no control for measuring student achievement and there needs to be an overall plan. Council replies that its method for monitoring the progress of the students included weekly student contact, monitoring class attendance and grades, and contact with instructors and counseling as needed. The weakness is rejected.
Lastly, the reviewer indicated that there was no plan in place or formulated for assuring student success. Council notes that close monitoring and weekly contact is the best assurance that it could provide and that it is impossible to assure success. Also, the program provided assistance through tutoring, financial counseling, and a variety of other services, all of which were designed to promote the completion of an educational program. On the basis of the record, the weakness is rejected.
It is concluded that substantial error is present in this rating. Accordingly, the rating of adequate by Reviewer #2 is raised from 7 points (high adequate) to 8 points (good).
Next, Council raises the same objection to the rating by Reviewer #3 who noted, as did Reviewer #2, that more information about past programs would have been helpful in order to use its success rates in prior programs as evidence that the proposed program had a likelihood of achieving improvement in teaching and student success. Past student success in a similar program is evidence of the likelihood of improvement in student success in Council’s proposed program. The student success rate has, however, no relevance regarding the likelihood of achieving improvement in teaching. Hence, the student-success aspect of this weakness is rejected and the teacher-improvement aspect is upheld.
Under this category, Reviewer #3 cited three weaknesses and only part of one weakness has been rejected. This is insufficient to constitute significant error and, accordingly, no change in the rating is appropriate.
Reviewer #4 opined that the proposal did not suggest any procedure to enhance survival in a selected program other than funding, child care, and others to motivate a student to go to class. Council adds that tutoring, counseling, job shadowing and mentoring are other means to enhance survival. Given the broad range of proposes services, there is no evidence in the record to support this weakness. It is rejected. Under this category, Reviewer #4 articulated five strengths of minimal merit and three weaknesses of some merit. In this context, the rejection of one weakness does not reflect a significant error in the rating. Hence, Reviewer #4’s rating of adequate is upheld.
Part C: Quality of the Project Design. In this category, the Secretary considers the design of the project which includes the actual details of the project, whether it addresses the needs of the target population, may be useful to others, provides linkages with other agencies providing services to the target population, and provides for feedback and improvements during its operation. Reviewer #4 rated Council’s project at 11 of 25 points which is the high end of inadequate. Reviewers #3, #2, and #1 graded the project at 14, 15, and 16 points, respectively. These scores reflect a mid adequate rating.
Reviewer #1 articulated extensive deficiencies in the project design category. While the project contemplates efforts to monitor student progress by the education counselor, the reviewer indicated that the plan does not describe how continuous improvement in the program will be implemented. Council’s response is an exhaustive list of events and activities that it maintains provide for continuous improvement of the project. However, its list consists of its normal activities. While these events may provide information, they are not specifically designed to implement changes -- the concern of the reviewer. The rating is upheld.
The Secretary utilizes one expansive and very detailed factor and seven limited factors in evaluating projects under this category. One limited factor, the quality of the methodology to be employed in the proposed project, was addressed by Reviewer #2. The reviewer related the sequence of events for handling a student such as a determination of needs, assistance, monitoring, etc. to assure the continued success as examples of the quality of the methodology. The reviewer articulated two weaknesses under this category. First, the absence of a long term plan for continuous improvement. Second, a major concern that there was a high level of resources dedicated to management support in contrast to student services.
Council disagrees with these weaknesses. Since the proposal does not have a long-term plan for continuous improvement, the first weakness is upheld. As to the second weakness, it is also upheld based on the discussion below.
All reviewers voiced significant concerns about the high cost of the program relative to the number of students and that the level of resources dedicated to management support was high in contrast to student services. One reviewer noted that administrative cost were 20% of the budget compared to the instructional cost for students at 2.5% of budget. Others indicated that costs were excessive for the number of students trained. These comments and assessments were noted under Part C: Quality of the Project Design and Part F: Adequacy of Resources.
Council’s response is twofold. First, it argues that tuition is not a significant aspect of the budget because a majority of its clients are eligible for Federal assistance which covers most of the costs. In addition, the percentage of the budget attributable directly to or on behalf of the students rises to 55% if one considers other student service/support such as mileage, childcare, books, class stipend, tutoring, and lab and registration fees. Second, one reviewer’s estimate of the cost per student at $6,000 was incorrect because it was calculated using 72 students, a figure reflecting the estimated number of students who would complete the program. Using the estimated number of students who would begin the program (90 students), the cost per student decreases to a more manageable figure of $4,600.
The reviewers’ concerns reflect subjective judgments formed after reviewing the objective facts such as the budget figures and the nature of the proposed project. These conclusions were reached by persons with relevant experience in the area and represent a reasonable assessment of the facts. Moreover, as circumstance would have it, they reached the same conclusion that, in itself, also suggests that such a conclusion was reasonable. Accordingly, the second weakness relating to the quality of methodology by Reviewer #2 is upheld and his or her mid adequate rating under this category is upheld.
Council questions only one of the weaknesses cited by Reviewer #3. Under the criterion addressing whether the design of the project reflects up-to-date knowledge from research, Council stated that the Adult Vocational Training staff subscribes to vocational training publications for ideas. Reviewer #3 indicated that this was a strength but that it should have been developed more thoroughly by indicating specific publications. Council responds by citing comments by Reviewer #1 and #2; however, these comments are focused on a different matter, not the use of publications for new ideas. Hence, the record supports this cited weakness. It also appears that the staff described by Council is not its staff but rather the staff at the schools with which it deals. If this was the case, then this matter was not appropriately cited as a strength in the first place. In either event, an increase in the assigned score is not warranted.
Reviewer #4 assigned a rating of 11 points or high inadequate in this category and articulated seven weaknesses. The reviewer criticizes the plan because it permits students to obtain a certificate of completion rather than a degree noting that there is a big difference between the two in terms of quality education. Council argues that the parameters of the grant program permits certificate and /or associates degrees. While this is true, a program that leads to an associates degree provides a better education than one that only leads to a certificate of completion. As such, a reasonable person could conclude that a program that includes a certificate of completion as an option has a weakness. The comment is upheld.
To the extent that the design of the program will lead to possible replication of its activities or strategies, Reviewer #4 noted that there was no strategy or guidance for possible replication of project activities. Council’s response cites comments by Reviewers #2 and #3. These comments note the presence of various means to collect information that could ultimately be of assistance in the replication of the project. Reviewer #4’s point dealt with a different concern, namely, the absence of a plan to disseminate publicly the project design or other relevant information. The comment is upheld.
Reviewer #4 wrote that the design of the proposal helps students to gain new skills and a job but the use of up-to-date knowledge is not reflected in any section in the proposal. Council argues that this is incorrect by citing comments made by Reviewer #2 who came to a different conclusion. This is a situation in which two reviewers disagree in their analysis based upon the information submitted by the applicant. The only relevant information in the application under this criterion was (1) that the Adult Vocational Training staff subscribes to unnamed vocational training publications to keep up-to-date and some ideas are adaptable to their program and (2) the needs for the project were determined in part from a state employment publication. This limited information supports clearly the conclusion reached by Reviewer #4. Council also cites Reviewer #1 who acknowledged the representation in the application that the Adult Vocational Training staff subscribes to some professional publications which may have contributed ideas. The problem here, as explained above, is that the program affected by these new ideas is not Council’s program. The programs belong to the vocational schools with which it deals and, as such, may not be considered under this criterion for purposes of this application.
Lastly, Council disagrees with Reviewer #4’s comment regarding the high level of resources devoted to management support in contrast to the student services. This issue was discussed above with respect to all reviewers and resolved against Council. Supra text pp. 5-6. Accordingly, the comment is upheld
On the basis of the above discussion, Reviewer #4’s rating of high inadequate is sustained.
Part D: Quality of Project Services. The Secretary considers the quality of the service to be provided and set forth eight criteria to be considered. The Secretary examines these services in light of their appropriate need, current reflection of up-to-date knowledge, and likely impact on the intended recipients. The services should lead to improvement of vocational skills, academic achievement, and employment.
Reviewer #4 assigned a rating of high inadequate (11 of 25 points). Reviewers #1, #3, and #2 rated this category as low end adequate (12 points) and mid adequate (14 and 15 points), respectively.
Council questions one weakness determined by Reviewer #4. This reviewer recognized that 1˝ months was set aside for the initial training of the staff; however, the proposal did not mention subsequent professional development or the quality thereof. Council points to comments by other reviewers to the effect that professional development was provided and that the staff was required to attend classes in computers and program related courses. Reviewer #4 was looking for more detail than the generalizations relied upon by the other reviewers. This weakness reflects a subjective judgment for which there is support in the record and will, therefore, not be overturned.
Part E: Quality of Project Personnel. The Secretary awards a maximum of 15 points in the quality of project personnel category. Council’s application was rated at 7 points by Reviewers #1, #2, and #3. Reviewer #4 assigned a rating of 8 points. Council disagrees with the low end adequate rating by the three reviewers.
Reviewer #1 and #3 shared a common concern that the project director should have a degree more advanced than an associate degree. Council argues that the proposed director has sufficient experience to oversee the operation citing her work experience including the directorship for the current IVEP program which was approved by the Department. The point of this weakness is not that she cannot perform the position. Rather, it is a view, based on philosophy and experience, that an individual with academic credentials more advanced than an associate degree will probably perform the various duties of this highly important position in a better manner. The weaknesses are upheld.
Reviewer #2 critiqued that the qualifications of the external consultants for the project appear to be weak in the area of consulting experience, assessment and evaluation. Additional documentation would be useful (resume, job description). Council acknowledges that resumes would have provided more information but were not provided because Council felt it was not necessary. The relevant criterion provides that the Secretary considers “[t]he qualifications, including relevant training and experience, of project consultants or subcontractors.” Council failed to satisfy the criterion as noted by Reviewer #2. The weakness is upheld.
In view of the above discussion under this category, there are no changes to the scores assigned by the reviewers.
Part F: Adequacy of Resources. Under this category, the Secretary considers the adequacy of resources for the proposed project. The maximum number of points awarded in this category is five. Reviewers #1, #3, and #4 assigned 2 points (high inadequate) and Reviewer #2 rated the proposal at 3 points (adequate).
Each reviewer had a concern that the project’s total cost was excessive based on the number of students trained and that the administrative costs were high compared to the student expenditures. These matters were addressed and resolved previously in the decision in the Quality of the Project Design category. Supra text pp.5-6. These weaknesses are upheld.
Reviewer #1 opined that the project will pay stipends but the applicant fails to describe how this will be accomplished and who will be responsible. Council notes that, in accordance with the job description, the education counselor will process the stipends and that the amount of the classroom stipends will be determined by a financial needs assessment. This is a minor point and, for the most part, addressed in the application. While it will be disregarded, it is not of such significance to warrant an increase in Reviewer #1’s rating. Reviewer #1’s rating remains 2 points, the same level as the ratings by Reviewer #3 and #4.
In summary, there are no changes to the ratings under this category.
Part G: Quality of the Management Plan. Under the quality of management plan category, the Secretary considers the adequacy of the management plan to achieve the objectives of the proposed project on time and within budget; whether time commitments of significant personnel are appropriate and adequate to meet the objectives of the proposed project; and whether there are procedures to ensure feedback and improvement in the operation of the project and mechanisms to ensure high quality products and services.
The reviewers rated this category in the adequate range (5 of 10 points by Reviewers #1 and #3, 6 points by Reviewer #4, and 7 points by Reviewer #2).
Reviewer #1described a multitude of strengths and weaknesses under the broad criterion addressing the adequacy of the management plan to achieve the objectives of the proposed project. One minor weakness was that the project did not breakdown training goals by occupation. Council explains that its experience in the early years of this program lead it to eliminate job specific goals because it limited the number of people it could help. Students have different aptitudes and areas of interest and, hence, the program must be flexible. This is an adequate explanation; however, it should have been included in the application. As such, the weakness must be sustained and no change in the assigned score by this reviewer is warranted.
In its application, Council presented a time-line graph indicating when various activities would take place. Reviewer #4 was concerned that the graph did not show the activities on a 3-year scale and doesn’t show when students will begin training. Council replies that this information is shown on the graph. The graph occupies 9 pages and is confusing to read and interpret. The information is, however, present in the graph and, given time, can be discerned. Hence, the weakness is not supported by the record and must be disregarded. Given that the other two weaknesses cited by this reviewer are minor in nature, like this weakness, the rejection of this weakness does not result in any significant error. Hence, no change in the score assigned by Reviewer #4 is warranted.
Part H: Quality of the Project Evaluation. Under this category, the Secretary requires appropriate methods of evaluation that measure the outcomes of the project, produce qualitative and quantitative data, provide performance feedback and permit periodic assessment of progress toward achieving the intended outcomes. Council has no objections to the assessments by the reviewers. The reviewers rated this category as high end inadequate (9 of 20 points by Reviewer #4) and low end adequate (10 points by Reviewers #1, #2, and #3).
Part I: Economic Development Plan. The Secretary proposed to award up to 10 additional points to those applications that “propose exemplary approaches that involve, coordinate with, or encourage tribal economic development plans.” 66 Fed. Reg. 564. Council does not dispute the adequate to good ratings by the reviewers (5 of 10 points by Reviewer #4, 6 points by Reviewer #1, and 7 points by Reviewers #2 and #3).
In summary, Council’s exceptions are generally without merit. Under Part B, the final rating by Reviewer #2 is adjusted from 7 to 8 points. As such, the final overall score by Reviewer #2 is 80 points. The overall scores by Reviewers #1, #3, and #4 remain unchanged at 71, 73, and 63 points, respectively. The adjustment to the final rating by Reviewer #2 will not significantly affect Council’s position vis-a-vis the award of a grant and, therefore, ED is not required to recompute and standardize Council’s score.
In light of the above, it is HEREBY ORDERED that the appeal by the Inter-Tribal Council, Inc. is dismissed with prejudice.
Allan C. Lewis
Chief Administrative Law Judge
Issued: July 24, 2001
 Council cited a weakness designated as “quality of instruction and instructors not addressed as being of high quality.” Council did not designate the reviewer to whom this weakness is attributable. An extensive review of the reviewers’ technical review forms failed to disclose this individual. In light of these circumstances, Council’s objection will not be addressed.