IN THE MATTER OF Docket No. 01-19-O
Wellpinit School District, Indian Vocational Education Program Proceeding
Samuel Mahaffy, Wellpinit, Washington, for the Wellpinit School District.
Daphna Krim, Esq., of the Office of the General Counsel, United States Department of Education, Washington, D.C., for the Assistant Secretary for Vocational and Adult Education.
Before: Allan C. Lewis, Chief Administrative Law Judge
The Wellpinit School District (Wellpinit) seeks a review of a decision by the Secretary not to include Wellpinit among the 30 applicants selected in a competitive process for the award of a grant under the Native American Vocational and Technical Education Program. 66 Fed Reg. 560 (2001). Wellpinit asserts that the Office of Vocational and Adult Education (ED) improperly employed a standardization program to determine the order of rank of the applicants and disputes various evaluation scores assigned by the reviewers to its application. In particular, Wellpinit seeks the elimination of the entire score by Reviewer #2 in determining its final ranking. Based on the analysis, infra, the employment of a standardization process is upheld. While Reviewer #2’s final raw score is not disregarded in its entirety, it is increased from 94 to 98 points. No adjustment to the final ratings assigned by the other reviewers is warranted.
The Native American Vocational and Technical Education Program provides grants to improve vocational and technical education programs that are consistent with the purposes of the Carl D. Perkins Vocational and Applied Technology Education Act. Pub. L. No. 88-210, 77 Stat. 403 (1963), as amended by the Carl D. Perkins Vocational and Applied Technology Education Amendments of 1998, Pub. L. No. 105-332, 112 Stat. 3076 (1998) (to be codified at 20 U.S.C. § 2301 et seq.). In awarding grants and entering into contracts, the Secretary shall ensure that the grants and contracts will improve vocational and technical education programs. 20 U.S.C. § 2326(e). Pursuant to this mandate, the Secretary published on January 3, 2001, a notification that solicited applications for a limited number of grants. 66 Fed. Reg. 560 (2001). The Secretary received 73 applications and, presently, anticipates funding 30 applications based upon their standardized scores.
ED employed eight panels, each comprised of four reviewers, to evaluate the applications. Each panel reviewed approximately nine applications. The applications were evaluated based upon eight criteria, totaling 125 points, and two additional criteria, totaling 15 points, as authorized under Section 116(e) of the Carl D. Perkins Vocational and Applied Technology Education Act. These criteria were set forth in the official notification in the Federal Register.
Wellpinit’s application was reviewed by a panel of four individuals and received raw scores of 103, 94, 110, and 127. These scores were standardized and then averaged. Wellpinit had a standardized average score of 101.5 and was ranked 33rd among the 73 applicants. This was 3 places removed from the last funded applicant that had a standardized score of 106.5.
A. RATING PROCEDURE
Based on its raw scores, Wellpinit’s average was 108.5 and was ranked 27th. After the standardization process, its average was 101.5 points and its order of ranking fell to 33rd , outside the funded group. As a result, Wellpinit argues that the appropriate method for ranking applicants should be based on their average raw scores rather than their standardized scores.
Initially, Wellpinit disputes the Department’s legal authority to employ a standardization method. It notes that the grant notice sets forth the program criteria and the maximum score for each criterion; however, it fails to indicate that the raw scores will be adjusted by the Department through a standardization formula. As such, Wellpinit asserts that the only rating approach authorized is the raw score method.
The legal authority to employ a standardization method is 34 C.F.R. § 75.217(c) (2000). Under 34 C.F.R. § 75.217, the Secretary describes how applications for competitive grants are selected. Subsection (c) provides that “[t]he Secretary prepares a rank order of the applications based solely on the evaluation of their quality according to the selection criteria.” It is readily apparent that individuals evaluating an application will not necessarily assign the same score to the various criteria because each reviewer will perceive, on occasion, different strengths and weaknesses and assign different scores. Wellpinit’s application is a typical example. The scores assigned to each criterion by the reviewers varied on most occasions. This reviewer bias also extends to panels that evaluate a limited number of applications within one competition. A statistical formula is used to eliminate the bias. This allows a subsequent ranking of the order of applications to be made based solely on the quality of the applications. Hence, standardization is a necessary act that must be performed by ED in order to comply with subsection (c) of the regulations.
Next, Wellpinit questions whether the statistical formula does, in fact, correct the bias and urges that it is arbitrary. The same argument was rejected in In re Sisseton Wahpeton Community College, Dkt. No. 95-86-O, U.S. Dept. of Education (Aug. 7, 1995). There, the applicant in the competitive Native American grant program argued that the use of more than one panel resulted in an erroneous ranking since different individuals formed the various panels. A true curve for purposes of comparison could only be obtained, argued the applicant, if all applications were read by the same reader. The tribunal accepted the statistical formula as a corrective tool when it dismissed the applicant’s argument—
ED responds that the scores of the applicants are subjected to a statistical standardization program designed to eliminate the potential problem of employing more than one panel to evaluate the applications. According to Susan Ahmed, the Chief Statistician at the National Center for Education Statistics, the mathematical basis for standardization process is—
statistically sound. It is based on a widely-accepted formula for standardizing scores from different distributions [and] . . . allows applications read by different panels to be ranked according to their relative merit regardless of what particular panel read any specific application.
ED Ex. F at 1-2. In light of the standardization program employed, there is no merit to College’s argument.
Sisseton Wahpeton at 4; Accord Inter-Tribal Council, Inc., Dkt. No. 01-20-O, U.S. Dept. of Education (July 24, 2001) at 2.
Wellpinit argues that ED’s submission of the Ahmed declaration from the Sisseton Wahpeton case has no relevance in this matter because there is no evidentiary basis laid to demonstrate that the circumstances in the two cases are similar. Moreover, this declaration was submitted in response to a different application years ago and responds specifically to allegations made by another applicant to which Wellpinit is not privy. Wellpinit’s points are correct in part. Ahmed’s declaration was not made with respect to this case. It was made, however, with respect to the same formula because the formula addressed in Sisseton Wahpeton was also used to standardize the current group of applications. (Berry Supp. Aff. at para. 11, 13.) The precedent of Sisseton Wahpeton
controls the determination in the present case in accordance with the doctrine of stare decisis.
B. OBJECTIONS TO SCORES ASSIGNED BY THE REVIEWERS
Wellpinit raises two broad concerns. First, it seeks to raise the scores of the reviewers on the theory that they created a requirement, i.e. the submission of resumes, that was not an official element under the “quality of personnel” criterion. Second, it argues that the rating by Reviewer #2 should be disregarded in its entirety for a variety of reasons.
Initially, the primary function of the tribunal at this stage of the award process is—
not to reevaluate the application and supplant the judgment of the reviewers with its judgment. The role of this tribunal is to ensure that the process of selecting successful grant recipients is fundamentally sound. Regarding the scoring aspect, it was held in Sisseton Wahpeton Community College at 6, that the score rendered by a reviewer with respect to each criterion should not be disturbed unless significant error exists. Such a standard recognizes that reasonable persons may differ in their evaluations and yet permits correction in the event of significant error.
In re Miccosukee Corporation, Dkt. No. 97-9-O, U.S. Dept. of Education (Mar. 24, 1997) at 5.
The issue concerning the absence of resumes arises under the “quality of project personnel” criterion. Here, the Secretary awards a maximum of 15 points and the applicants address the quality of the personnel who will carry out the proposed project including—
(2) In determining the quality of project personnel, the Secretary considers the extent to which the applicant encourages applications for employment from persons who are members of groups that have traditionally been underrepresented based on color, national origin, gender, age, or disability.
(3) In addition, the Secretary considers the following factors:
(i) The qualifications, including relevant training and experience, of the project director.
(ii) The qualifications, including relevant training and experience, of key project personnel, especially the extent to which the project will use instructors who are certified to teach in the field in which they will provide instruction.
(iii) The qualifications, including relevant training and experience, of project consultants or subcontractors.
Wellpinit’s application was rated at the high end of inadequate by Reviewer #2 (6 out of 15 points), at the high end of adequate by Reviewers #1 and #4 (10 and 11 points, respectively), and good by Reviewer #3 (12 points).
Wellpinit asserts that these reviewers believed, erroneously, that resumes or job descriptions were a required attachment to the application. As a result, they cited the absence of resumes as deficiencies in 9 instances and assigned scores that were improperly low. These comments constituted error on the part of the reviewers because the Department’s notice to the public was silent regarding a requirement to attach resumes to the application and the notice should have affirmatively stated that the submission of resumes was required as part of the materials to be considered in evaluating an application. According to Wellpinit, such affirmative notice is mandated because it is the Department’s practice to include within its notices such a statement and, more importantly, the inclusion within the notice is necessary in order to make this aspect a part of a selection element.
ED responds, in effect, that Wellpinit’s argument misses the point. The reviewers’ comments regarding the absence of resumes were made as part of their critiques. Since the “quality of project personnel” criterion focuses on the qualifications of key project personnel including relevant training and experience, ED maintains that it was not “the absence of that particular type of document [that constituted a weakness according to a reviewer] but rather . . . [the weakness was] the absence of the information necessary to address the criterion’s elements . . . information that is often if not typically conveyed in professional resumes.” ED Br. at 9. In the context of this competition, resumes were neither required nor intended to be required. Hence, the notice did not state that they were required and did not instruct applicants specifically to include them. Also, ED notes that, for this reason, the instructions to reviewers during orientation did not include an instruction to look for resumes, as such.
Initially, ED is correct that this criterion focuses upon the qualifications of key project personnel that includes relevant training and experience and has no requirement mandating the submission of resumes. It is clear from the comments and their contexts that each reviewer was not citing the absence of a resume document as a deficiency. Rather, each reviewer was concerned that the application lacked detail regarding the qualifications of various personnel and the reference to a resume or job description was the manner in which they articulated this deficiency. The comments by the reviewers are addressed below.
Wellpinit stated in its application that “the project director will be the superintendent of the Wellpinit School District who has more than 25 years experience in Indian education, considerable involvement in economic development issues with several tribes, and is nationally recognized as a leader in the field of Indian education.” App. at 13. Reviewer #4 was looking for more detail as he/she commented that “no resume or detailed job description [was] contained in the proposal.” Similarly, Reviewer #1 noted that “there was no resume to verify the education or experience” of this individual.
The proposed project included a subcontractor to perform evaluations. Wellpinit described his qualifications in a vague and limited manner as—
an experienced social researcher who has worked extensively with intervention effectiveness research. He has depth of experience in traditional effectiveness evaluation and has designed innovative qualitative evaluation tools for measuring shifts in attitudes and expectations and their subsequent impact on behaviors, especially in socio-economically diverse communities.
App. at 15. Hence, Reviewer #1 commented that an evaluator was mentioned but “there was no resume or information provided on this person.” Reviewer #2 opined that “this reader found no resume on this person.”
As to the qualifications including relevant training and experience of Wellpinit’s project consultants or subcontractors, Reader #2 stated—
[a]gain, there was no way to determine this criteria. Individuals are not named, resumes are not included. There is no way for this reader to determine the qualifications, relevant training, and experience.
Wellpinit’s application included 3 full time key personnel besides the project director and 3 part time positions—
[a]dditional staff will include a technology teacher, a natural resource teacher, and a hospitality and health careers teacher. A half time counselor will be engaged experienced in the theoretical and practical aspects of effective mentoring, and developing effective student support functions. The superintendent of the district will allocate 20% of FTE to serve in consulting capacity. An experienced researcher and evaluation expert will contribute 20% FTE to designing and implementing effective evaluation protocols and coordinating communication with the Advisory Council for the new program.
App. at 12-13. The reviewers cited the absence of detailed information as they stated “[a]gain, this reader found no resume on these people” (Reader #2) and “no job descriptions or resume included in the proposal” (Reader #4).
Reader #3 summed up the absence of information in this part of the application as “the overall weakness in this category is that the applicant does not provide resumes for persons being proposed for leadership and staff roles in the new program.”
The tribunal finds no significant error in the ratings assigned by the reviewers, especially the high end of inadequate rating by Reviewer #2.
Next, Wellpinit seeks to purge the score assigned by Reviewer #2 as part of its overall rating on the theory that this reviewer (1) materially and substantially failed to follow the written instructions, (2) failed to follow various criteria, including the definition of what constitutes a vocational-technical program, and (3) failed to exercise due care and diligence in grading the application. These allegations are based upon Wellpinit’s disagreement with 22 weaknesses cited by Reviewer #2, a majority of the critical comments made by this reviewer.
Wellpinit’s first broad category of errors is that Reviewer #2 failed to follow the technical review guidelines because the reviewer commented, as a weakness, that the application did not address certain sub-indicators. This comment was made in 13 instances and, according to Wellpinit, was incongruent with the directions in the orientation materials that reviewers were to “provide substantial comments . . . . We ask that you fill your pages with detailed comments that strictly address the selection
criteria . . . . One line comments are not acceptable . . . . Comments should be relevant and specific.” As a result, Wellpinit maintains that these comments were not specific and relevant but vague and not substantive. As such, they should be disregarded.
ED disagrees and addresses the 13 instances pointing out, inter alia, that the comments were appropriate because the information in the application did not address a specific sub-indicator or element of a criterion, that an additional comment was present that further amplified this remark, or that a comment was not material as the reviewer struck it from the weakness column or moved it into the strength column. ED also notes that many of these weaknesses were recognized by the other reviewers and that the differences in scoring among the reviewers are attributable to their subjective evaluations, a recognized and accepted aspect of the rating process.
The tribunal reviewed each of these weaknesses cited by Reviewer #2 and the related, pertinent aspects of Wellpinit’s application. The comments were relevant regarding the elements in question. The three comments addressed below are representative of this group.
Under the “quality of project services” criterion, element (v) addresses the nature, extent, and likelihood that professional development for staff members would lead to improvements in their duties. Reviewer #2 stated as a weakness that “this specific sub-indicator [(v)] was not addressed in the project.” (Comment 11.) This comment was succinct, on point, and reflects the record.
Under the same criterion, element (viii) requires the applicant to describe the likelihood that the services will lead to improvements in the skills necessary to gain employment or build capacity for independent living. Under the weakness column, Reviewer #2 commented that “given the limited information provided, this [element] is difficult to determine.” (Comment 13.) The target population of Wellpinit’s project was non college bound 7th through 12th graders and adults. While the program proposed 4 skill/curriculum areas (life skills, technology, natural resource planning and hospitality and health), the curriculum in these broad areas was not addressed or developed in the application. Hence, the reviewer’s point was appropriate as well as supported by the record.
In a similar fashion, Reviewer #2 critiqued that Wellpinit failed to address element (vi) under the adequacy of resources criterion. (Comment 19.) This element sought information regarding the potential for continued support of the project after Federal funding ends. Wellpinit’s application ignored this element. The comment was apt and supported by the record.
A second broad category of objections concerns 3 comments (nos. 1, 12, and 20) by Reviewer #2 that Wellpinit feels are inaccurate or not reflective of a careful reading of the proposal. The first comment related to element (ii) under the “need for project” criterion. This element provides that the Secretary considers—
[t]he magnitude of the need for the . . . proposed project (as evidenced by data such as the labor market demand, occupational trends, surveys, recommendations from accrediting agencies, or tribal economic development plans).
Reviewer #2 wrote as a strength that, currently, career and technical education services do not exist in this school district. This would be consistent with a high magnitude of need for the services proposed. As a weakness, the reviewer indicated that there were no surveys or data cited in the application to substantiate the magnitude of need.
Wellpinit argues that Reviewer #3 found that it clearly demonstrated a case for need. This reply does not cite the presence of surveys or data in the application that substantiates the magnitude of its need. Hence, it is not responsive. Reviewer #2’s observation is an accurate reflection of the record. It is upheld.
The second comment concerned element (vi) of the “quality of project services” criterion. Reviewer #2 opined that the project provided no employment statistics for the career-technical education activities talked about (future job openings, pay, etc.). Wellpinit asserts, citing a comment by Reviewer #3, that “data related to high levels of unemployment . . . identify a need to have this program.” There is no error because the reviewers are addressing two different topics -- the availability of job opportunities and the rate of unemployment.
The third comment addressed element (i) under the “quality of the management plan” criterion. Element (i) measures “[t]he adequacy of the management plan to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, timelines, and the milestones and performance standards for accomplishing project tasks.” Reviewer #2 noted that the “project does not have a management plan. (timeline by objective, person responsible, cost).” Wellpinit disagrees and, as support, cites a comment by Reviewer #4 that the “administration appears adequate for insuring that high quality products and services are delivered.” Once again, Wellpinit’s citation does not address the point raised by Reviewer #2.
A third broad category of errors alleged by Wellpinit is that 6 comments by Reviewer #2 are evaluative, but not based on criteria authorized by the Secretary and, therefore, should be disregarded. Of the 6 comments, 3 comments (nos. 14, 15, and 16) concerned the resume issue under the “quality of project personnel” criterion and were addressed above. The tribunal found that these comments by Reviewer #2 were appropriate. Supra text at 5-7. The remaining 3 comments are dealt with below.
Two comments (nos. 17 and 18) addressed elements (ii) and (iv) of the “adequacy of resources” criterion. Element (ii) provides that the proposal must show—
the relevance and demonstrated commitment (e.g., articulation agreements, memoranda of understanding, letter of support, commitments to employ project participants) of the applicant, members of the consortium, local employers, or tribal entities to be served by the project, to the implementation and success of the project.
In comment no. 17 under element (ii), Reviewer #2 noted that the application did not list any “non-federal funds [to be contributed by the applicant or others] and given the lack of information, this reader could not determine this criteria.” Wellpinit asserts that the RFP had no requirement for matching or non-federal funds. ED responds that the comment is appropriate.
While a financial commitment to the project by an applicant, a tribal entity, or a local employer clearly evidences a demonstrated and relevant commitment, element (ii) focuses on non financial types of commitment such as a commitment to employ a participant from the program or an articulation agreement. The tribunal agrees with Wellpinit that this comment falls outside the parameters of element (ii) and, therefore, must be disregarded.
Under element (iv), the Secretary considers “the extent to which the costs are reasonable in relation to the objectives, design, services, and potential significance of the proposed project.” Reviewer #2 noted, in comment no. 18, that project costs are not broken out by objectives, services, etc. Wellpinit asserts that there is no such requirement. The tribunal agrees. The weakness will be disregarded. Inasmuch as 2 of the 3 weaknesses noted by Reviewer #2 under the “adequacy of resources” criterion have been disregarded, it is appropriate to increase his/her rating from 4 (high adequate) to 5 (good to excellent), the highest rating. Thus, Reviewer #2’s raw score is increased by 1 point.
The third and last comment (no. 6) addresses the project design. Under elements (i) and (ii) of the “quality of the project design” criterion, the Secretary considers the extent to which goals, objectives, and outcomes are clearly specified and measurable and the extent to which the design of the proposed project is appropriate to, and will address, the needs of the target population or other identified needs. Reviewer #2 commented adversely that—
[t]he design is talked about on pages 9 & 10. While the 4 core skill/curriculum areas are laid out—this reader does not think that this constitutes a CTE [career-technical education] program. This reader was looking for Industry based standards, placement #’s, wages ladders, completion rates etc. and did not find it.
Wellpinit argues vociferously that this comment reflects Reviewer #2’s belief about what constitutes vocational and technical education and that this belief was inconsistent with its statutory definition of this term. Moreover, Wellpinit maintains that—
it is not a leap to conclude that this reviewer rejected the merits of this application out-of-hand, and judged the application as weak because of his/her perception that there was not an appropriate definition of a CTE program.
Br. at 5. Wellpinit adds that—
[t]he low evaluator [Reviewer #2] whose comments reflected a divergent perspective on what constitutes voc-tech education gave a scoring of 16 out of a possible 25 points in this category. If the low reviewers scoring in this category were raised to the average reviewers scoring of 22, the six-point addition to this score would have raised the ranking of the applicants standing within the range of funded applications.
Br. at 7.
ED maintains that Reviewer #2’s comment does not indicate or infer that a personal definition was applied by the reader. The term “CTE” is a commonly used term in the vocational education field that is used interchangeably with the term vocational and technical education that is defined in 20 U.S.C. § 2302(29). According to ED, Reviewer #2 was looking for information “missing” from the application such as industry based standards that are addressed within the definition of vocational and technical education in 20 U.S.C. § 2302(29), placement and completion rates that are examples of the goals and objectives element of criterion (c)(2)(i), and wage ladders that are measures of goals, objectives and outcomes of vocational training. Br. at 18. ED concludes that this reader gave Wellpinit an overall “adequate” score under the quality of design criterion and did not introduce any inappropriate elements into his/her analysis.
Under 20 U.S.C. § 2302(29), the term “vocational and technical education” means organized educational activities that–
(A) offer a sequence of courses that provides individuals with the academic and technical knowledge and skills the individuals need to prepare for further education and for careers (other than careers requiring a baccalaureate, masters’s or doctoral degree) in current or emerging employment sectors; and
(B) include competency-based applied learning that contributes to the academic knowledge, higher-order reasoning and problem-solving skills, work attitudes, general employability skills, technical skills, and occupation-specific skills, of an individual.
The question of whether a proposed project constitutes vocational and technical education, as it must be under the grant program, is a mixed question of fact and law. This question should be determined by ED and be so determined before the applications are submitted to the rating panels. It is definitely not within the province of the reviewers to consider, let alone make this determination. Yet, this is what the reviewer did. He/she considered Wellpinit’s curriculum in the context of his/her understanding of the nature and scope of a CTE program. This constitutes error and is significant because, as viewed by Reviewer #2, this would constitute a fundamental flaw in the quality of the project’s design.
Reviewer #2 assigned a score of 16 points out of a possible of 25 points under the “quality of the project design” criterion. This was a mid adequate rating since the point range within adequate varied from 12 to 18 points. Among the elements under the quality of the project design, the basic design of the program is most significant Other factors under this criterion such as replication, coordination with others, and performance feedback are important but to a much lesser degree. Regarding these latter elements, Reviewer #2 articulated mixed reviews. In terms of the overall design, Reviewer #2 noted that, as a strength, the goals, objectives and specific aims were explained; however, as a weakness, the discussion lacked specific numbers and baseline information that could measure or reflect the extent of success of the program, such as anticipated completion rates, placement numbers, wage ladders, etc. In this context, an adjustment of 3 points is warranted. The rating by Reviewer #2 under the “quality of the project design” criterion is changed from 16 to 19 points which moves the rating from mid adequate to good. Based upon this adjustment and the 1 point adjustment under the “quality of personnel” criterion, the final raw score by Reviewer #2 is increased from 94 to 98 points.
Lastly, there is no justification in the record to disregard the rating by Reviewer #2 in its entirety, as urged by Wellpinit.
Wellpinit’s argument is based upon three categories of errors. As determined above, no errors were present among the 16 comments questioned by Wellpinit under two categories -- the failure to follow the technical review guidelines and the failure to read carefully the application. Under the third category, the failure to employ the established criteria, Wellpinit disputed 6 comments. Three comments were determined to be proper. The other 3 comments were determined to be in error and resulted in a 4 point adjustment to the overall rating by this reviewer. While these determinations do not warrant the purging of his/her evaluation, one matter remains to be addressed.
The lynchpin of Wellpinit’s purging argument is its belief that Reviewer #2’s evaluation under the criteria was affected by his/her belief that the project was not a CTE type program. Excluding the quality of project design criterion, the other criteria address elements unrelated to this issue. His/her adverse comments under the other criteria were specific, focused on the pertinent elements, and supported by the record. There is no indication that his/her judgement was affected and no basis to grant the relief sought by Wellpinit.
C. ALTERNATIVE GROUNDS FOR FUNDING
Lastly, Wellpinit raises several points in an effort to obtain funding even though the final rating of its application was insufficient to place it within the 30 applications approved for funding. Under the notice, the Secretary “may select other than the most highly rated applications for funding if doing so would . . . create a more equitable distribution of funds under this competition among Indian tribes, tribal organizations, Alaska Native entities, or eligible Bureau-funded schools.”
Wellpinit urges that funding its application would result in an equitable distribution of funds since (1) it has never received funding under this program, (2) there are no vocational and technical education programs currently available on the Spokane Indian Reservation, and (3) there is no meaningful access to such education programs outside the reservation due to its geographic rural remoteness. Moreover, its high score under the 10 point special economic development plan criterion (38 of 40 points by the 4 reviewers) and that the services under the grant will be provided within a federally designated Empowerment Zone/Enterprise Community are additional factors that support its selection for special treatment.
ED responds that the limited amount of funds precludes funding all applicants and that, as explained by the individual responsible for choosing the successful applicants under the program, the Department decided that there was no need or justification for funding an applicant out of the rank order because the thirty highest scoring applicants represented a broad range of eligible entities serving numerous tribes and locations. Moreover, ED asserts that there is no legal basis or justification for giving Wellpinit any additional advantage simply because it qualified for additional points under the economic plan development criterion or that it is located within an empowerment zone/enterprise zone.
The question is whether the Secretary abused his discretion when he chose to fund the top 30 ranked applications to the exclusion of Wellpinit’s application. This is a competitive program. The 30 ranked competitors proposed better programs as measured under the grant criteria. Also, these 30 applicants reflected a broad diversity of tribes and locations. These factors constitute a rational basis for the decision and, accordingly, there is no abuse of discretion. The decision will not be disturbed.
In light of the above, it is HEREBY ORDERED that the Assistant Secretary for Vocational and Adult Education determine forthwith the ranking of Wellpinit School District’s application among its competitors in accordance with this decision.
Allan C. Lewis
Chief Administrative Law Judge
Issued: August 8, 2001
 Lastly, Wellpinit argues that the standardization formula resulted in one applicant’s final rating exceeding the maximum points permitted under the guidelines in the notice. Therefore, the formula contradicts the guidelines and must be disregarded. Wellpinit misses the point. The point specific guidelines are used in determining the raw scores and the adjusted average points derived under the standardization formula are used to rank the applicants. Hence, there is no contradiction and the guidelines are not violated.
 “Sub-indicator” was a term employed by this reviewer to refer to a specific, detailed aspect of the proposed project that the Secretary considered as a critical element under a criterion.
 Comment nos. 2-4, 7-11, 13, 19, and 21-23 in Wellpinit’s memorandum.
 Two comments (nos. 4 and 8) are not relevant in this proceeding. Comment 4 is considered as a strength because the reviewer moved it from the weakness column into the strength column. Comment 8 was crossed out by the reviewer and, therefore, was disregarded by the reviewer.
 Fifty CTE programs were reviewed in developing the design and thus, it reflected up-to-date research and knowledge under element (iv). Replication of the project by others was considered very likely under element (iii). In contrast, the application did not address the quality of the methodology to be employed under element (viii) and coordination and linkages with others (elements (v) and (vi)) could only be inferred based on working with other school districts and the dissemination efforts discussed in the application.