APPLICATION OF THE


MESCALERO APACHE TRIBE,

DECISION

Appearances:



The Mescalero Apache Tribe (Tribe) submitted a request for a hearing to appeal a decision by the Secretary not to include Tribe among the 25 applicants selected in a competitive grant process for the award of a grant under the Indian Vocational Education Program. 34 C.F.R.
§ 401.23 (1996). Tribe's application was one of 80 applications reviewed, evaluated, and rated.
Based on the findings and analysis set forth infra, the review process was proper and only a nominal adjustment in the score assigned to Tribe's application is warranted. This adjustment will not affect the decision by the Secretary not to include Tribe among the successful applicants for an award of a grant.

I. OPINION

The Indian Vocational Education Program provides financial assistance to projects that provide vocational education for the benefit of Native Americans. 34 C.F.R. § 401.1. In a June 19, 1996, notice, the Secretary invited eligible tribal organizations to submit applications for a competitive grant which were to be evaluated using the selection criteria and corresponding weights designated by 34 C.F.R. § 401.21, as supplemented by 34 C.F.R. § 401.20(b) and 34 C.F.R. § 401.20(e). 61 Fed. Reg. 31,364. As a result of this notice, 83 applications were received for fiscal year 1996. The Department deemed three applications ineligible, and, therefore, 80 applications were evaluated for grant funding. The Department selected the top 25 eligible applications for grant funding and the lowest ranking successful applicant had a standardized score of 82.15.See footnote 11/ Tribe's application was given a standardized score of 19.81 and was ranked 80th of the 80 applications evaluated.See footnote 22/

Tribe raises one procedural matter and many substantive exceptions regarding the purported weaknesses in its application as reported by its three reviewers. The procedural matter will be addressed first.

Tribe maintains that its application was not given an in-depth review. It asserts that the evaluation notes of two of the three reviewers were especially difficult to read which it attributes to a hastily conducted review. In its view, the overall impression of the reviewers' comments indicates that the reviewers were hurried, rushed, and somewhat confused as they addressed and assigned points to the various rating criteria addressed in its proposal. Tribe attributes this cursory review to the fact that the reviewers had 83 applications to examine and score.

ED responds that Tribe misunderstands the nature of the rating process. Tribe's application was rated by one of twelve panels used by the Department to review the 80 applications. ED reports that the panels worked on a full-time basis over a five-day period to review the 80 applications and that each panel consisted of three individuals with relevant expertise and included one Native American. Therefore, in ED's view, there is no basis for concluding that the review process was anything other than thoughtful, fair, and thorough.

The Tribe's application was given adequate consideration. Each panel reviewed seven to eight applications over a five-day period and, therefore, it appears that approximately one-half of one day was spent reading and evaluating each application. The comments by Tribe's three reviewers reflect that each reviewer had read and evaluated the Tribe's application as it pertained to each rating category. While their comments were not necessarily written in complete sentences, this was obviously due, in part, to the limited space on the review sheet. Nevertheless, the comments addressed the strengths and weaknesses in the various categories as perceived by the reviewers and reflected a genuine effort to evaluate the application. Given these circumstances, Tribe's application received an adequate review.

Tribe raises numerous objections to the comments by the reviewers regarding the weakness of its application in an effort to raise its overall rating. Its overall standardized score was 19.81 which reflected the average of the three standardized scores of 17.68, 21.47, and 20.29 by the reviewers. Thus, in order to move within the rank of the funded applicants, Tribe's standardized score must be raised over 60 points.

In its brief, ED assigned, as a general matter, the various objections by Tribe to one of several general categories. One category, for example, was that Tribe's objection plainly missed the point of the reviewer's comment. The effect of this overly generalized approach is that ED failed, in many instances, to address the specific merits of Tribe's objections. On other occasions, ED failed to respond to an objection by Tribe. Such omissions will be considered as concessions.

When there is an express or implied concession by ED of error by a reviewer or a determination of error by the tribunal, such error must be measured against the standard of review to determine whether a reviewer's score warrants an adjustment. An adjustment is proper if “there is significant error.” In re Sisseton Wahpeton Community College, Dkt. No. 95-86-O, U.S. Dept. of Education, (Aug. 7, 1995) at 6.

Initially, Tribe argues that the reviewers rated its application “low” in most categories because they failed to comprehend that the grant program was part of an effort to expand an existing program funded by Tribe. ED responds that there is no legal basis for any special consideration for the applicant due to these circumstances and suggests, in any event, that the reviewers were possibly aware of this circumstance. Tribe does not explain the manner in which this circumstance affected its scores and, therefore, there is no substance to its argument. Moreover, the tribunal cannot perceive any partiality in the criteria or an evaluation thereunder whether the application proposes a new program, the continuation of an existing program, or the expansion or modification of an existing program. Therefore, this argument is rejected. In addition, Tribe's concern about the effects of its rural location and its “hands on” approach to vocational training is also misplaced. The applications are evaluated based upon criteria which are clearly defined and equally applicable in all of these circumstances.

Tribe raises various concerns regarding each of the criteria evaluated by the reviewers. These concerns are addressed below.

Part A: The reviewers rated the program factors category as mid-level inadequate (5 of 25 points by reviewers A and C) and the upper range of inadequate (8 points by reviewer B). The reviewers raised several weaknesses which were not challenged by Tribe. The weaknesses were noted primarily in the description of the goals of the project and were also present in the topics dealing with the targets of the program and the program approach addressing the selection, assessment/feedback and the curriculum of the project.

Regarding the goals of the program, the reviewers B and C raised concerns that the goals were too general and not measurable based on the information provided. In addition, reviewer A noted that there was no mention of anticipated graduation or placement rates.

Tribe maintains that there were measurable goals in that the students had to satisfy the standards of the Job Opportunities and Basic Skill Building Trades Qualification Standards, the Job Training Partnership Act Building Trade Skill Standards, the appropriate union apprenticeship program standards, and the GED final exam or the graduation requirements of the local high school. ED failed to address this response. The Tribe is correct in that it referred to the standards in these programs; however, without further amplification of the standards or the capabilities of an individual who satisfies these standards, mere reference thereto does not provide the reviewers with measurable goals.

Regarding the absence of graduation and placement rates, Tribe argues that various statements in its application reflect that successful students are assured a position in the building trades or closely related fields. ED responds that the Tribe's response misses the point.

ED is correct in part. Tribe did not address the absence of an estimated graduation rate in its application. Such a rate is meaningful in that it would force Tribe to make a realistic assessment of its proposed program. Tribe did address, however, the placement question although it was not sufficient to remove it as a weakness. The sense of the reviewer's comment and other comments is that there should be a better showing of the number of available slots in each field of study offered under the program as well as an indication of the number of actual employment positions available. Thus, the record supports the determinations by the reviewers.

The selection criteria and potential target candidates were described by reviewers A and C as vague and the curriculum, as detailed in the application, was considered weak and vague and needed more expansion according to reviewers B and C. In response, Tribe indicates that the target recruits are members of the Mescalero Apache Tribe and that, as outlined in its application, the selection standard for candidates was based upon the “probability of completion [of the program] and need.” It also explained that only a brief summary of the curriculum was included in the application in order to simplify matters for the reviewers.

ED argues that Tribe relies upon new material in responding to the weakness regarding the selection criteria for potential candidates. Even though ED's response misses the point, the record contains sufficient evidence for the reviewer to conclude that the criteria was vague and, therefore, this determination is upheld.

Regarding the weak curriculum determination by reviewers B and C, ED maintains that Tribe's response should be disregarded because it addresses a matter in which opinions may differ. Tribe's response actually explained its justification for the generalized approach and did not dispute the fact that the curriculum, as outlined in the application, was vague. Under the program factors criteria, an application is evaluated, in part, on whether it “[i]ncludes a thorough description of the approach . . . including [the] . . . [c]urriculum.” 34 C.F.R. § 401.21(a)(6). Thus, the determination by the reviewers will not be altered.

As a result of the above, no modification of the scores under the program factors category is warranted.

Part B: Of the 15 points available in the need category, reviewers A and C graded the application at 5 points (upper end of inadequate) and reviewer B assigned a 7 point rating
(mid-level of adequate).

Reviewers A and C emphasized a primary point that there was no link between jobs, training, and current skills. Reviewer A noted the absence of any mention of current education and skill level of the proposed applicants and Tribe's failure to articulate whether the job market was static, growing, or declining. Reviewer C added that the data lacked depth and did not convincingly establish a need for the project.

Tribe's responds, in effect, that the link is the Advisory Committee which will determine the specific aspects of the training, establish the required curriculum, and oversee and evaluate the program. It also notes that the members of the Advisory Committee, as individuals, will be the employers of the successful candidates.

Tribe's response adds as well as significantly amplifies upon the information presented in its application and, therefore, this additional information was not available to the reviewers. The reviewers were also concerned with the apparent lack of depth in the development and interrelationship among the jobs, training, and current skills of the targeted individuals. These latter weaknesses are judgment determinations made by the reviewers. Given the evidence in the record, it is sufficient to support these determinations and, therefore, they will not be disturbed.

Reviewer B indicated that ongoing and planned activities were vague in nature and that Tribe's 17% unemployment rate raised concern about the severity of its need for the program.

Tribe's exception to the vague description of its activities is not responsive as it indicates that the businesses, which will be the potential recipients of the program graduates, will have an influence on the type of training and changes in its format to accommodate their needs. Accordingly, this weakness is upheld.

Regarding the severity of its need for a program, Tribe emphasizes that 17% unemployment is, in its view, a very high figure, which leaves many families without skilled breadwinners. ED argues that the degree of an unemployment problem is a matter of judgment upon which the parties may disagree and, therefore, the weakness should be sustained. The tribunal agrees that this is a judgment matter and will not disturb the reviewer's determination.

Part C: The reviewers scored the plan of operation category consistently in that reviewers A and C rated the category as 5's, i.e. the upper end of the inadequate , and reviewer B rated the category as a 6, i.e. the lower end of adequate. The reviewers reported few strengths and many weaknesses. Tribe's objections addressed only 1 (no placement goals) of 4 weaknesses noted by reviewer A; 3 (objectives lacked detail explaining high wage jobs, failed to include statistics for anticipated enrollment, completion, and placement; no placement goals; and education committee has no apparent direct relationship with the tribal education department) of 5 weaknesses detailed by reviewer B; and the 4 weaknesses indicated by reviewer C (no specific management plan; timelines are vague and only for 1 year; no data on available resources to accomplish the goals; insufficient info to conclude project will be well organized and managed).

ED failed to address Tribe's objections regarding reviewer C's purported weakness that there was an absence of a specific management plan and reviewer A's and B's weakness that there were no placement goals. As a result, these weaknesses constitute errors by the reviewers.

ED argues that two objections (one by reviewer C for the absence of timelines for the activities such as hiring, program preparation, student recruitment and selection, instruction, and evaluation and job placement and the other by reviewer B for the absence of a direct relationship between the education committee and the tribal education department) represent differences of opinion between the reviewer and Tribe and, therefore, as qualified evaluators, their scores should not be altered.

The determination of the purported absence of timelines by reviewer C has no support in the application. Clearly and in detail through a chart (table 5), Tribe set forth the specific month or months in which the various activities pertaining to its program would take place. This determination constitutes error by reviewer C. Regarding reviewer B's view that there is no apparent direct relationship between the education committee and the tribal education department, Tribe admits there is no direct link between the two and that communications are directed through the Office of the Tribal President. Thus, there is no difference of opinion in this matter and the weakness as determined by reviewer B stands.

Reviewer C noted that, under the plan of operation, there was no data on available resources. Tribe replies that the available resources are listed on page 11 as the Advisory Committee which is a committee comprised of representatives of tribal businesses and established to oversee the program including the provision of field projects and the materials needed therefor.

ED includes this objection into the general category of objections that plainly miss the point of the reviewer's comment and, beyond this generalization, ED fails to articulate the manner in which it reached this general conclusion.

ED's response, without more, is meaningless. One of the rating criteria under the plan of operation category is that the applicant indicate the manner in which it “intends to use its resources and personnel to achieve each objective” of the project. 34 C.F.R. § 401.21(c)(3). Tribe made an effort to identify its sources of manpower and, to a more limited extent, its sources for the required building and other materials necessary for the projects. Elsewhere in the application, there are dollars allocated for the purchase of tools, supplies, and similar items. The tribunal finds that the reviewer C's comment, given the overall information supplied by the Tribe, may not be considered as a weakness.

Reviewer B indicated that the statement of objectives would have been stronger if the applicant had added statistics on anticipated enrollment, completion, and placement and had explained the term high wage jobs in the sentence: “[t]he benefit that will result from this project is the full- time employment of up to forty individuals in relatively high wage jobs.”

Tribe excepts, in part, and explains, in effect, that the vocational training envisioned by the project would enable the successful participants to work in trade positions where the wage rates were substantially in excess of the minimum wage.

ED maintains that the Tribe's objection reflects a difference of opinion with the reviewer and, therefore, there is no basis for its objection.

Tribe's exception does not involve a difference of opinion as ED asserts. It simply explained a term whose meaning is readily apparent from the information in the application. Since this information was available to the reviewer B, it cannot be considered a weakness.

Reviewers A and B had only 1 weakness out of their 4 or 5 comments which was in error. In this situation, no change in their scores is warranted. On the other hand, reviewer C had three of his/her 4 weaknesses rejected by the tribunal. This represents significant error and reviewer C's score in this category is adjusted from a 5 to a 7 -- the latter being the mid-level of adequate.

Part D: The reviewers rated the category of key personnel at the upper end of adequate (reviewers A and C assigned 6 of a possible 10 points) and slightly below excellent (reviewer B assigned 9 points). Reviewer A noted two weaknesses one of which was the lack of depth in the resume of the project director, John Shendo. The sole weakness reported by Reviewer B was the brief “bio” of John Shendo. Reviewer C wrote that his resume failed to detail specific experience and, therefore, it was difficult to determine the extent and level of his experience.

Tribe maintains that the resume of Mr. John Shendo was prepared by a very professional organization and has been presented to and accepted by various organizations and businesses.

ED responds that Tribe's objection raises simply a difference of opinion with the reviewers and, therefore, is far from being proof of a significant error.

The tribunal agrees with ED in that whether, and to what extent, the resume of the project director constitutes a weakness in this category is a judgment determination by the reviewers. As a general rule, judgment determinations will not be disregarded. In this case, there is support for the determination in the record and, therefore, the weakness will not be disturbed.

Part E: The reviewers rated the budget and cost effectiveness category as inadequate (reviewer A assigned 1 point out of 5 points) and adequate (reviewer C assigned 2 points and reviewer B assigned 3 points). The reviewers noted the absence of explanations and justifications for the amounts assigned to various expenses in the budget such as office and student academic supplies ($1,000 per month), expendable student shop supplies ($1,000 per month per student), small tools for training ($380 per student), student training tool kits ($320 per student), telecommunications ($800 per month), vehicles ($28,920), and the $5,000 pay differential between the skill instructor position and the academic position.

Tribe agrees that there should have been better justification for the office and student academic supplies, expendable student shop supplies, small tools for training, and student tool kits. It argues that the cost assigned to the telecommunication item was justified given the small phone company in the area whose rates are high. The $28,920 cost for vehicles was justified in that vehicle costs are expensive in its area and this cost purportedly included the mileage to and from the work sites. It explains the $5,000 higher salary for the skill instructor position over the academic instructor position on the basis that more importance was given to the position of a trade instructor than an academic instructor.

ED argues that, with respect to the unexplained salary differential between the skill instructors and the academic instructor, Tribe offered the explanation in its submission to the tribunal and, therefore, this reflects additional information which was not available to the reviewers.

It is reasonable that an unexplained differential in salary may reflect a weakness in the application. The proffered explanation, which otherwise would eliminate this weakness, was made too late for the reviewers to consider. In this circumstance, the score of the only reviewer to cite this point as a weakness (reviewer B) will remain, therefore, unchanged.

ED did not address Tribe's objection regarding the absence of a justification for the purportedly unreasonable vehicle expense and the purportedly excessive phone budget. Therefore, the tribunal concludes that ED concedes error in these matters. These matters represented only one (the phone budget) of seven weaknesses noted by reviewer A, one (the vehicle) of three weaknesses determined by reviewer B, and one (the phone) of two weaknesses noted by reviewer C. Given their respective scores and the weaknesses noted by each reviewer, these concessions are insufficient to constitute significant error which would justify a change in any scores.

Part F: The reviewers rated the evaluation plan category at the upper end of inadequate (reviewers A and B assigned 3 points out of a possible 10 points) and the lower end of inadequate (reviewer C assigned 1 point). The reviewers noted deficiencies such as the lack of a formative evaluation plan, the means to collect data, and the absence of provisions for formative feedback and improvements in the program.

Tribe acknowledges that its application contains no details regarding the manner in which appropriate data will be collected. Tribe urges, however, that the program will be continuously monitored by the director, the advisory committee, and other members of the Tribe's professional community. In addition, Tribe indicates that a formative evaluation will be performed by Eastern New Mexico University.

ED responds that the Tribe's articulation of the monitoring bodies and the designation of the university as the formative evaluator reflects additional information not included in the application and, therefore, it was not available for the reviewers to evaluate. As such, ED concludes that the reviewers did not err in their scores.

The tribunal agrees with ED. The application was deficient in the manner described by the reviewers. Additional information cannot be provided at this point to alter its scores in a category.

Part G: The reviewers rated the employment opportunity category at the upper end of inadequate (reviewers A and C assigned 5 points out of a possible 20 points) and mid-level adequate (reviewer B assigned 10 points). Reviewers A, B, and C viewed the letters of commitment to hire graduates as too general and containing only inferential employment commitments. According to the reviewers, the letters lacked firm commitments to hire a specific number of graduates with specified skills. Reviewer A also indicated there was no mention of placement goals and employment opportunities off the reservation.

Tribe only addresses the problem with the letters of commitment and argues that the letters indicate a strong commitment. ED responds that this matter reflects a difference in opinion in interpreting the letters between the reviewers and Tribe.

A review of the letters indicates that there is evidence which supports the determinations by the reviewers and, therefore, the scores will remain unchanged.

Part H: Tribe was given an adequate score (2, 3, and 2 out of 5 points) under the economic development criterion. The weaknesses, according to the reviewers, were the application's failure to articulate how its project involves, coorindates with, and encourages the Tribe's updated Overall Economic Development Plan and its Strategic Economic Development Plan which was being finalized. The mere reference to these plans was not enough in the eyes of the reviewers.

Tribe's response is that these plans are “available for review.” In ED's view, this is not sufficient as all pertinent information must be included with the application. The tribunal agrees with ED. The facts surrounding the program and its interrelationship with other programs must be provided in the application. The reviewers may not look beyond the application for information pertaining to the program.

In summary, Tribe's exceptions are generally without merit. Other than under Part C, the errors by the reviewers did not constitute significant error and warrant an adjustment in their respective scores under each criterion. Thus, the overall scores assigned by reviewers A and B are sustained. The overall score by reviewer C requires a 2 point upward adjustment as noted in the discussion under Part C -- the plan of operation. This adjustment will not significantly affect Tribe's position vis-a-vis the award of a grant and, therefore, ED is not required to recompute and restandardize Tribe's score.

II. ORDER

In light of the above, it is HEREBY ORDERED that the appeal by the Mescalero Apache Tribe is dismissed with prejudice.


Issued: April 1, 1997
    Washington, D.C.



______________    

SERVICE

______________



On April 1, 1997, a copy of the attached document was sent by certified mail, return receipt requested to the following:

John Shendo, Jr.
Tribal Education Director
Mescalero Apache Tribe
P.O. Box 176
Mescalero, New Mexico 88340

Daphna Crotty, Esq.
Office of the General Counsel
United States Department of Education
600 Independence Avenue, S.W.
FOB-10B, Room 5442
Washington, D.C. 20202-2110


Footnote: 11/ The evaluation was based upon the seven criteria set forth in 34 C.F.R. § 401.21 totaling 100 points and two additional criteria set forth in 34 C.F.R. § 401.20(e) totaling 10 points. See also 34 C.F.R. § 401.20(b). Accordingly, the maximum score was 110 points.
Footnote: 22/ The numerical scores assigned by the reviewers were standardized and averaged to eliminate the problem of employing more than one review panel to evaluate the applications.