APPLICATION OF THE
Appearances:
The Indian Vocational Education Program provides financial assistance to projects that provide
vocational education for the benefit of Native Americans. 34 C.F.R. § 401.1. In a June 19, 1996,
notice, the Secretary invited eligible tribal organizations to submit applications for a competitive
grant which were to be evaluated using the selection criteria and corresponding weights
designated by 34 C.F.R. § 401.21, as supplemented by 34 C.F.R. § 401.20(b) and 34 C.F.R. §
401.20(e). 61 Fed. Reg. 31,364. As a result of this notice, 83 applications were received for
fiscal year 1996. The Department deemed three applications ineligible, and, therefore, 80
applications were evaluated for grant funding. The Department selected the top 25 eligible
applications for grant funding and the lowest ranking successful applicant had a standardized
score of 82.15.See footnote 11/ Tribe's application was given a standardized score of 19.81 and was ranked
80th of the 80 applications evaluated.See footnote 22/
Tribe raises one procedural matter and many substantive exceptions regarding the purported
weaknesses in its application as reported by its three reviewers. The procedural matter will be
addressed first.
Tribe maintains that its application was not given an in-depth review. It asserts that the
evaluation notes of two of the three reviewers were especially difficult to read which it attributes
to a hastily conducted review. In its view, the overall impression of the reviewers' comments
indicates that the reviewers were hurried, rushed, and somewhat confused as they addressed and
assigned points to the various rating criteria addressed in its proposal. Tribe attributes this
cursory review to the fact that the reviewers had 83 applications to examine and score.
ED responds that Tribe misunderstands the nature of the rating process. Tribe's application was
rated by one of twelve panels used by the Department to review the 80 applications. ED reports
that the panels worked on a full-time basis over a five-day period to review the 80 applications
and that each panel consisted of three individuals with relevant expertise and included one Native
American. Therefore, in ED's view, there is no basis for concluding that the review process was
anything other than thoughtful, fair, and thorough.
The Tribe's application was given adequate consideration. Each panel reviewed seven to eight
applications over a five-day period and, therefore, it appears that approximately one-half of one
day was spent reading and evaluating each application. The comments by Tribe's three
reviewers reflect that each reviewer had read and evaluated the Tribe's application as it pertained
to each rating category. While their comments were not necessarily written in complete
sentences, this was obviously due, in part, to the limited space on the review sheet. Nevertheless,
the comments addressed the strengths and weaknesses in the various categories as perceived by
the reviewers and reflected a genuine effort to evaluate the application. Given these
circumstances, Tribe's application received an adequate review.
Tribe raises numerous objections to the comments by the reviewers regarding the weakness of its
application in an effort to raise its overall rating. Its overall standardized score was 19.81 which
reflected the average of the three standardized scores of 17.68, 21.47, and 20.29 by the reviewers.
Thus, in order to move within the rank of the funded applicants, Tribe's standardized score must
be raised over 60 points.
In its brief, ED assigned, as a general matter, the various objections by Tribe to one of several
general categories. One category, for example, was that Tribe's objection plainly missed the
point of the reviewer's comment. The effect of this overly generalized approach is that ED
failed, in many instances, to address the specific merits of Tribe's objections. On other
occasions, ED failed to respond to an objection by Tribe. Such omissions will be considered as
concessions.
When there is an express or implied concession by ED of error by a reviewer or a determination
of error by the tribunal, such error must be measured against the standard of review to determine
whether a reviewer's score warrants an adjustment. An adjustment is proper if “there is
significant error.” In re Sisseton Wahpeton Community College, Dkt. No. 95-86-O, U.S. Dept.
of Education, (Aug. 7, 1995) at 6.
Initially, Tribe argues that the reviewers rated its application “low” in most categories because
they failed to comprehend that the grant program was part of an effort to expand an existing
program funded by Tribe. ED responds that there is no legal basis for any special consideration
for the applicant due to these circumstances and suggests, in any event, that the reviewers were
possibly aware of this circumstance. Tribe does not explain the manner in which this
circumstance affected its scores and, therefore, there is no substance to its argument. Moreover,
the tribunal cannot perceive any partiality in the criteria or an evaluation thereunder whether the
application proposes a new program, the continuation of an existing program, or the expansion or
modification of an existing program. Therefore, this argument is rejected. In addition, Tribe's
concern about the effects of its rural location and its “hands on” approach to vocational training
is also misplaced. The applications are evaluated based upon criteria which are clearly defined
and equally applicable in all of these circumstances.
Tribe raises various concerns regarding each of the criteria evaluated by the reviewers. These
concerns are addressed below.
Part A: The reviewers rated the program factors category as mid-level inadequate (5 of 25 points
by reviewers A and C) and the upper range of inadequate (8 points by reviewer B). The
reviewers raised several weaknesses which were not challenged by Tribe. The weaknesses were
noted primarily in the description of the goals of the project and were also present in the topics
dealing with the targets of the program and the program approach addressing the selection,
assessment/feedback and the curriculum of the project.
Regarding the goals of the program, the reviewers B and C raised concerns that the goals were
too general and not measurable based on the information provided. In addition, reviewer A
noted that there was no mention of anticipated graduation or placement rates.
Tribe maintains that there were measurable goals in that the students had to satisfy the standards
of the Job Opportunities and Basic Skill Building Trades Qualification Standards, the Job
Training Partnership Act Building Trade Skill Standards, the appropriate union apprenticeship
program standards, and the GED final exam or the graduation requirements of the local high
school. ED failed to address this response. The Tribe is correct in that it referred to the
standards in these programs; however, without further amplification of the standards or the
capabilities of an individual who satisfies these standards, mere reference thereto does not
provide the reviewers with measurable goals.
Regarding the absence of graduation and placement rates, Tribe argues that various statements in
its application reflect that successful students are assured a position in the building trades or
closely related fields. ED responds that the Tribe's response misses the point.
ED is correct in part. Tribe did not address the absence of an estimated graduation rate in its
application. Such a rate is meaningful in that it would force Tribe to make a realistic assessment
of its proposed program. Tribe did address, however, the placement question although it was not
sufficient to remove it as a weakness. The sense of the reviewer's comment and other comments
is that there should be a better showing of the number of available slots in each field of study
offered under the program as well as an indication of the number of actual employment positions
available. Thus, the record supports the determinations by the reviewers.
The selection criteria and potential target candidates were described by reviewers A and C as
vague and the curriculum, as detailed in the application, was considered weak and vague and
needed more expansion according to reviewers B and C. In response, Tribe indicates that the
target recruits are members of the Mescalero Apache Tribe and that, as outlined in its application,
the selection standard for candidates was based upon the “probability of completion [of the
program] and need.” It also explained that only a brief summary of the curriculum was included
in the application in order to simplify matters for the reviewers.
ED argues that Tribe relies upon new material in responding to the weakness regarding the
selection criteria for potential candidates. Even though ED's response misses the point, the
record contains sufficient evidence for the reviewer to conclude that the criteria was vague and,
therefore, this determination is upheld.
Regarding the weak curriculum determination by reviewers B and C, ED maintains that Tribe's
response should be disregarded because it addresses a matter in which opinions may differ.
Tribe's response actually explained its justification for the generalized approach and did not
dispute the fact that the curriculum, as outlined in the application, was vague. Under the
program factors criteria, an application is evaluated, in part, on whether it “[i]ncludes a thorough
description of the approach . . . including [the] . . . [c]urriculum.” 34 C.F.R. § 401.21(a)(6).
Thus, the determination by the reviewers will not be altered.
As a result of the above, no modification of the scores under the program factors category is
warranted.
Part B: Of the 15 points available in the need category, reviewers A and C graded the
application at 5 points (upper end of inadequate) and reviewer B assigned a 7 point rating
(mid-level of adequate).
Reviewers A and C emphasized a primary point that there was no link between jobs, training,
and current skills. Reviewer A noted the absence of any mention of current education and skill
level of the proposed applicants and Tribe's failure to articulate whether the job market was
static, growing, or declining. Reviewer C added that the data lacked depth and did not
convincingly establish a need for the project.
Tribe's responds, in effect, that the link is the Advisory Committee which will determine the
specific aspects of the training, establish the required curriculum, and oversee and evaluate the
program. It also notes that the members of the Advisory Committee, as individuals, will be the
employers of the successful candidates.
Tribe's response adds as well as significantly amplifies upon the information presented in its
application and, therefore, this additional information was not available to the reviewers. The
reviewers were also concerned with the apparent lack of depth in the development and
interrelationship among the jobs, training, and current skills of the targeted individuals. These
latter weaknesses are judgment determinations made by the reviewers. Given the evidence in the
record, it is sufficient to support these determinations and, therefore, they will not be disturbed.
Reviewer B indicated that ongoing and planned activities were vague in nature and that Tribe's
17% unemployment rate raised concern about the severity of its need for the program.
Tribe's exception to the vague description of its activities is not responsive as it indicates that the
businesses, which will be the potential recipients of the program graduates, will have an
influence on the type of training and changes in its format to accommodate their needs.
Accordingly, this weakness is upheld.
Regarding the severity of its need for a program, Tribe emphasizes that 17% unemployment is, in
its view, a very high figure, which leaves many families without skilled breadwinners. ED
argues that the degree of an unemployment problem is a matter of judgment upon which the
parties may disagree and, therefore, the weakness should be sustained. The tribunal agrees that
this is a judgment matter and will not disturb the reviewer's determination.
Part C: The reviewers scored the plan of operation category consistently in that reviewers A and
C rated the category as 5's, i.e. the upper end of the inadequate , and reviewer B rated the
category as a 6, i.e. the lower end of adequate. The reviewers reported few strengths and many
weaknesses. Tribe's objections addressed only 1 (no placement goals) of 4 weaknesses noted by
reviewer A; 3 (objectives lacked detail explaining high wage jobs, failed to include statistics for
anticipated enrollment, completion, and placement; no placement goals; and education
committee has no apparent direct relationship with the tribal education department) of 5
weaknesses detailed by reviewer B; and the 4 weaknesses indicated by reviewer C (no specific
management plan; timelines are vague and only for 1 year; no data on available resources to
accomplish the goals; insufficient info to conclude project will be well organized and managed).
ED failed to address Tribe's objections regarding reviewer C's purported weakness that there
was an absence of a specific management plan and reviewer A's and B's weakness that there
were no placement goals. As a result, these weaknesses constitute errors by the reviewers.
ED argues that two objections (one by reviewer C for the absence of timelines for the activities
such as hiring, program preparation, student recruitment and selection, instruction, and
evaluation and job placement and the other by reviewer B for the absence of a direct relationship
between the education committee and the tribal education department) represent differences of
opinion between the reviewer and Tribe and, therefore, as qualified evaluators, their scores
should not be altered.
The determination of the purported absence of timelines by reviewer C has no support in the
application. Clearly and in detail through a chart (table 5), Tribe set forth the specific month or
months in which the various activities pertaining to its program would take place. This
determination constitutes error by reviewer C. Regarding reviewer B's view that there is no
apparent direct relationship between the education committee and the tribal education
department, Tribe admits there is no direct link between the two and that communications are
directed through the Office of the Tribal President. Thus, there is no difference of opinion in this
matter and the weakness as determined by reviewer B stands.
Reviewer C noted that, under the plan of operation, there was no data on available resources.
Tribe replies that the available resources are listed on page 11 as the Advisory Committee which
is a committee comprised of representatives of tribal businesses and established to oversee the
program including the provision of field projects and the materials needed therefor.
ED includes this objection into the general category of objections that plainly miss the point of
the reviewer's comment and, beyond this generalization, ED fails to articulate the manner in
which it reached this general conclusion.
ED's response, without more, is meaningless. One of the rating criteria under the plan of
operation category is that the applicant indicate the manner in which it “intends to use its
resources and personnel to achieve each objective” of the project. 34 C.F.R. § 401.21(c)(3).
Tribe made an effort to identify its sources of manpower and, to a more limited extent, its
sources for the required building and other materials necessary for the projects. Elsewhere in the
application, there are dollars allocated for the purchase of tools, supplies, and similar items. The
tribunal finds that the reviewer C's comment, given the overall information supplied by the
Tribe, may not be considered as a weakness.
Reviewer B indicated that the statement of objectives would have been stronger if the applicant
had added statistics on anticipated enrollment, completion, and placement and had explained the
term high wage jobs in the sentence: “[t]he benefit that will result from this project is the full-
time employment of up to forty individuals in relatively high wage jobs.”
Tribe excepts, in part, and explains, in effect, that the vocational training envisioned by the
project would enable the successful participants to work in trade positions where the wage rates
were substantially in excess of the minimum wage.
ED maintains that the Tribe's objection reflects a difference of opinion with the reviewer and,
therefore, there is no basis for its objection.
Tribe's exception does not involve a difference of opinion as ED asserts. It simply explained a
term whose meaning is readily apparent from the information in the application. Since this
information was available to the reviewer B, it cannot be considered a weakness.
Reviewers A and B had only 1 weakness out of their 4 or 5 comments which was in error. In this
situation, no change in their scores is warranted. On the other hand, reviewer C had three of
his/her 4 weaknesses rejected by the tribunal. This represents significant error and reviewer C's
score in this category is adjusted from a 5 to a 7 -- the latter being the mid-level of adequate.
Part D: The reviewers rated the category of key personnel at the upper end of adequate
(reviewers A and C assigned 6 of a possible 10 points) and slightly below excellent (reviewer B
assigned 9 points). Reviewer A noted two weaknesses one of which was the lack of depth in the
resume of the project director, John Shendo. The sole weakness reported by Reviewer B was the
brief “bio” of John Shendo. Reviewer C wrote that his resume failed to detail specific
experience and, therefore, it was difficult to determine the extent and level of his experience.
Tribe maintains that the resume of Mr. John Shendo was prepared by a very professional
organization and has been presented to and accepted by various organizations and businesses.
ED responds that Tribe's objection raises simply a difference of opinion with the reviewers and,
therefore, is far from being proof of a significant error.
The tribunal agrees with ED in that whether, and to what extent, the resume of the project
director constitutes a weakness in this category is a judgment determination by the reviewers. As
a general rule, judgment determinations will not be disregarded. In this case, there is support for
the determination in the record and, therefore, the weakness will not be disturbed.
Part E: The reviewers rated the budget and cost effectiveness category as inadequate (reviewer A
assigned 1 point out of 5 points) and adequate (reviewer C assigned 2 points and reviewer B
assigned 3 points). The reviewers noted the absence of explanations and justifications for the
amounts assigned to various expenses in the budget such as office and student academic supplies
($1,000 per month), expendable student shop supplies ($1,000 per month per student), small
tools for training ($380 per student), student training tool kits ($320 per student),
telecommunications ($800 per month), vehicles ($28,920), and the $5,000 pay differential
between the skill instructor position and the academic position.
Tribe agrees that there should have been better justification for the office and student academic
supplies, expendable student shop supplies, small tools for training, and student tool kits. It
argues that the cost assigned to the telecommunication item was justified given the small phone
company in the area whose rates are high. The $28,920 cost for vehicles was justified in that
vehicle costs are expensive in its area and this cost purportedly included the mileage to and from
the work sites. It explains the $5,000 higher salary for the skill instructor position over the
academic instructor position on the basis that more importance was given to the position of a
trade instructor than an academic instructor.
ED argues that, with respect to the unexplained salary differential between the skill instructors
and the academic instructor, Tribe offered the explanation in its submission to the tribunal and,
therefore, this reflects additional information which was not available to the reviewers.
It is reasonable that an unexplained differential in salary may reflect a weakness in the
application. The proffered explanation, which otherwise would eliminate this weakness, was
made too late for the reviewers to consider. In this circumstance, the score of the only reviewer
to cite this point as a weakness (reviewer B) will remain, therefore, unchanged.
ED did not address Tribe's objection regarding the absence of a justification for the purportedly
unreasonable vehicle expense and the purportedly excessive phone budget. Therefore, the
tribunal concludes that ED concedes error in these matters. These matters represented only one
(the phone budget) of seven weaknesses noted by reviewer A, one (the vehicle) of three
weaknesses determined by reviewer B, and one (the phone) of two weaknesses noted by reviewer
C. Given their respective scores and the weaknesses noted by each reviewer, these concessions
are insufficient to constitute significant error which would justify a change in any scores.
Part F: The reviewers rated the evaluation plan category at the upper end of inadequate
(reviewers A and B assigned 3 points out of a possible 10 points) and the lower end of
inadequate (reviewer C assigned 1 point). The reviewers noted deficiencies such as the lack of a
formative evaluation plan, the means to collect data, and the absence of provisions for formative
feedback and improvements in the program.
Tribe acknowledges that its application contains no details regarding the manner in which
appropriate data will be collected. Tribe urges, however, that the program will be continuously
monitored by the director, the advisory committee, and other members of the Tribe's
professional community. In addition, Tribe indicates that a formative evaluation will be
performed by Eastern New Mexico University.
ED responds that the Tribe's articulation of the monitoring bodies and the designation of the
university as the formative evaluator reflects additional information not included in the
application and, therefore, it was not available for the reviewers to evaluate. As such, ED
concludes that the reviewers did not err in their scores.
The tribunal agrees with ED. The application was deficient in the manner described by the
reviewers. Additional information cannot be provided at this point to alter its scores in a
category.
Part G: The reviewers rated the employment opportunity category at the upper end of inadequate
(reviewers A and C assigned 5 points out of a possible 20 points) and mid-level adequate
(reviewer B assigned 10 points). Reviewers A, B, and C viewed the letters of commitment to
hire graduates as too general and containing only inferential employment commitments.
According to the reviewers, the letters lacked firm commitments to hire a specific number of
graduates with specified skills. Reviewer A also indicated there was no mention of placement
goals and employment opportunities off the reservation.
Tribe only addresses the problem with the letters of commitment and argues that the letters
indicate a strong commitment. ED responds that this matter reflects a difference in opinion in
interpreting the letters between the reviewers and Tribe.
A review of the letters indicates that there is evidence which supports the determinations by the
reviewers and, therefore, the scores will remain unchanged.
Part H: Tribe was given an adequate score (2, 3, and 2 out of 5 points) under the economic
development criterion. The weaknesses, according to the reviewers, were the application's
failure to articulate how its project involves, coorindates with, and encourages the Tribe's
updated Overall Economic Development Plan and its Strategic Economic Development Plan
which was being finalized. The mere reference to these plans was not enough in the eyes of the
reviewers.
Tribe's response is that these plans are “available for review.” In ED's view, this is not sufficient
as all pertinent information must be included with the application. The tribunal agrees with ED.
The facts surrounding the program and its interrelationship with other programs must be
provided in the application. The reviewers may not look beyond the application for information
pertaining to the program.
In summary, Tribe's exceptions are generally without merit. Other than under Part C, the errors
by the reviewers did not constitute significant error and warrant an adjustment in their respective
scores under each criterion. Thus, the overall scores assigned by reviewers A and B are
sustained. The overall score by reviewer C requires a 2 point upward adjustment as noted in the
discussion under Part C -- the plan of operation. This adjustment will not significantly affect
Tribe's position vis-a-vis the award of a grant and, therefore, ED is not required to recompute
and restandardize Tribe's score.
In light of the above, it is HEREBY ORDERED that the appeal by the Mescalero Apache Tribe
is dismissed with prejudice.
Issued: April 1, 1997
Washington, D.C.
John Shendo, Jr.
Tribal Education Director
Mescalero Apache Tribe
P.O. Box 176
Mescalero, New Mexico 88340
Daphna Crotty, Esq.
Office of the General Counsel
United States Department of Education
600 Independence Avenue, S.W.
FOB-10B, Room 5442
Washington, D.C. 20202-2110