Summary of Research

The FNS Research Corner provides summaries of recently completed and ongoing research conducted by the U.S. Department of Agriculture’s Food and Nutrition Service (FNS) in the area of child nutrition. For further information, contact the Office of Analysis, Nutrition, and Evaluation (OANE) at (703) 305-2117. Links to published studies and reports, as well as to descriptions of ongoing studies conducted by OANE, are available from the FNS Web site at http://www.fns.usda.gov/oane/.

Evaluation of the National School Lunch Program (NSLP) Application/Verification Pilot Projects: Impacts on Deterrence, Barriers, and Accuracy
The U.S. Department of Agriculture (USDA) sponsored the NSLP Application/Verification Pilot Projects to test the ways in which it could improve the process for certifying students for free or reduced-price meals. This report presents findings on the impact of two alternatives on the current application-based certification process: Up-Front Documentation and Graduated Verification. These were tested in 12 public school districts over a three-year period.

Background
Millions of U.S. children participate in the National School Lunch Program (NSLP) each day and receive free or reduced-price lunches that make important contributions to their overall nutrition. Concerns have mounted, however, that many of the children eligible for free or reduced-price meals may, in fact, be ineligible because their family income is too high. Under the existing eligibility process, families are required to state their income on the application for benefits but do not need to submit any additional documentation. Districts select a small sample of applications for income verification, which is done later in the year.

To address the question of whether the eligibility process could be made more accurate, the USDA sponsored pilot projects testing two new approaches to certifying eligibility: Up-Front Documentation and Graduated Verification. Districts using Up-Front Documentation require families to document their income or receipt of public assistance at the time they submit their application for free or reduced-price school meals. Districts then use this documentation to make an eligibility determination, but do not verify approved applications later in the school year. Districts using Graduated Verification allow families to use a modified version of the standard application process. Although this process does not require income documentation, the verification method includes an initial validation of a small sample of approved applications. The districts continue the validation process if 25% or more of the applications in the initial test result in benefit reduction or termination.

Study Design and Methodology
The study used a comparison design to select additional districts that did not participate in the three-year pilots, but had similar economic characteristics and geographic locations. Researchers then compared the two types of districts to estimate the impact on the accuracy of the certification process, as well as the degree that it deterred ineligible families or discouraged eligible families from applying. Data for the study came from administrative records provided by the districts, and telephone and in-person interviews with approximately 3,000 households with children enrolled in the study districts during Fall 2002.

Key Findings
Deterrence of Ineligible Families
Neither Up-Front Documentation nor Graduated Verification resulted in an observable deterrence of erroneous certifications. The rates of erroneous certification among ineligible students were less than 5% in Up-Front Documentation comparison districts and less than 10% in Graduated Verification comparison districts. Neither Up-Front Documentation nor Graduated Verification had a statistically significant negative effect on the rate of erroneous certifications. In other words, neither pilot had a statistically significant deterrent effect.

Barriers for Eligible Families
Both sets of pilot procedures caused barriers among some eligible students. Rates of certification among each group of eligible students examined were lower in pilot districts than in comparison districts. Some of these differences were statistically significant, indicating that Up-Front Documentation and Graduated Verification led to increased barriers among eligible students.

Accuracy Among Certified Students
Compared to current procedures, neither set of pilot procedures changed certification accuracy at a level that could be detected in the study. Overall, about 18% of students certified for free meals were ineligible for the benefits they were receiving. However, the estimated impacts of Up-Front Documentation and Graduated Verification on certification accuracy were small and not statistically significant.

School Food Authority Administration of National School Lunch Program Free and Reduced-Price Eligibility Determination
In an effort to understand the accuracy of the current eligibility determination system, FNS carried out a review of 14 large school food authorities (SFAs) in 2002, which included approximately 3,500 applications selected for verification. Each of the SFAs had enrollments between 24,000 and 178,000 children in 2001-02. In total, these 14 SFAs enroll approximately 950,000 children, more than half of who were approved for free and reduced price-meals as of October 31, 2001.

This report provides the results of the review of SFA administrative records regarding applications, verifications, and meal ticket status. Data collection was conducted by FNS Regional Office staff members. The report examines the:

• Results of the verification process in 14 large SFAs;
• Accuracy with which SFAs conduct the application and verification processes within the National School Lunch Program (NSLP);
• Accuracy of the meal ticket status at the end of the 2001-02 school year of children selected for verification; and
• Rate at which children, whose benefits are terminated in the verification process, reapply for free and reduced-price meal benefits following termination.

Key Findings
Verification Results

• Overall results. On average, 43% of applications verified in study districts had no changes to their benefit levels as a result of the verification process, while 55% had a benefit reduction/termination. Two percent had an increase in benefits from reduced-price to free.
• Respondent ineligibility. Among respondents to verification, 21% had a benefit reduction or termination. In random sampling districts, 14% of respondents had a benefit reduction or termination. Focused sampling districts resulted in 40% of respondents receiving a benefit reduction or termination. Four percent of respondents to verification had a benefit increase from reduced-price to free meal.
• Reasons for benefit change. Non-response to the verification process accounted for most of the applications that received reduced or terminated benefits. Seventy-seven percent of benefit reductions/terminations in focused sampling districts were due to non-response, as compared to 80% in random sampling districts. Random sampling SFAs had a higher rate of benefit increase from reduced-price to free (3%) than focused sampling SFAs (1%).

Results from Income Applications Versus Categorical Applications

• Income and household size applications. Among applications selected for verification that were approved on the basis of income and household size, 57% had benefits reduced or terminated. Among respondents, 23 had their benefits reduced or terminated. The average rate of benefit reduction or termination among respondents was lower in random sampling districts (18%) than in focused sampling districts (34%).
• Categorically approved applications. Categorically approved applications are where households report a Food Stamp/Temporary Assistance for Needy Families/Food Distribution Program on Indian Reservations (FS/TANF/FDPIR) case number as the basis of their eligibility. This type of application was less likely to have a benefit reduction or termination as a result of the verification process. Among categorically approved applications, there was a 30% rate of benefit reduction/termination, as compared to a 57% rate among income and household size applications.

Accuracy of SFA review of applications and verifications

• Application review. On average, 94% of applicants that reported household income and size and received initial approvals by SFAs for free and reduced price meal benefits were accurate based on the information provided, while 6% of initial approvals were inaccurate.
• Verification determinations. Ninety-two percent of the verification decisions by SFAs were appropriate and consistent with the documentation provided by parents. Eight percent of the verification decisions were inappropriate.
• Meal ticket status. Among children whose applications and verifications were reviewed, the end of year meal ticket status was accurate in 83% of the cases and inaccurate in 17% of cases.
• Direction of inaccuracy. More than 75% of all SFA administrative errors identified in this study resulted in households being approved for a greater level of benefit than was supported by the materials provided. Less than 25% of all errors resulted in households being approved for a lower level of benefit than to which they were entitled.

Reapplication of non-responders to verification
Twenty-nine percent of households that had benefits terminated in the verification process reapplied later in the 2001-02 school year. On average, 93% of such reapplications were approved for free or reduced-price meal benefits, while 7% were denied.

The following section provides a brief description of ongoing FNS research and the current status of these studies.

Evaluation of the School Breakfast Program Pilot Project
The FNS is currently evaluating the congressionally mandated three-year School Breakfast Pilot Project that began in school year 2000-01. The six participating school districts, selected from among 386 districts that applied to participate, are:

• Harrison County School District, Gulfport, MS
• Independent School District of Boise City, Boise, ID
• Santa Rosa City Schools, Santa Rosa, CA
• Shelby County Board of Education, Columbiana, AL
• Washington Elementary School District, Phoenix, AZ
• Wichita Public Schools, Wichita, KS

School breakfasts were available at no cost to all elementary school students regardless of their household income (universal-free school breakfasts) in about half of the pilot schools evaluated within each of the six school districts listed above. The remaining schools continued to operate the regular School Breakfast Program. This evaluation is rigorously assessing the effects of the availability of universal-free school breakfasts on breakfast participation and a broad range of student outcomes, including academic achievement, school attendance and tardiness, classroom behavior and attentiveness, and dietary intakes. It is also examining how school districts implement universal-free breakfasts, including its effect on cost, paperwork, and other administrative requirements.

Initial findings, based on first-year data, were published in an interim report released in November 2002 (FNS Research Corner, Spring 2003). The final evaluation report, which will be available in 2004, will include analyses based on first-year data, as well as that collected during the second and third year of the pilot. The data collection in the second and third year was generally limited to the collection of health and disciplinary data from schools, and student and school level administrative data, including attendance, tardiness, breakfast participation, and scores on academic achievement tests. During the third year, another round of implementation-related interviews and site visits was conducted to identify any changes that occurred in the methods used by schools to provide universal-free breakfasts.

Case Study of Verification Outcomes in Major Metropolitan Areas
As the National School Lunch Program Application/Verification pilot evaluation did not include large school districts, FNS sought to learn about certification accuracy in large school districts through a study of 21 SFAs, located in seven large metropolitan areas (Boston, Los Angeles, Minneapolis-St. Paul, Orlando, Salt Lake City, Tulsa, and Virginia Beach-Norfolk-Newport News). The objectives were to assess the outcomes of the verification process and to independently determine the income eligibility category of two groups of households: 1) a sample of household that did not respond to verification and are, therefore, subject to losing eligibility for benefits and 2) a sample of households whose benefit levels remained unchanged during the verification process. Determining the eligibility status of each of these groups provides a useful measure of the effectiveness of verification in detecting and reducing inaccuracy among sampled certifications.

The study recorded the verification outcome of all cases selected during Fall 2002 for the study SFAs, and conducted in-home interviews with samples of non-respondent households and households whose benefits remained unchanged. The study also collected information on the rate of re-approval among households that did not respond to verification, but were initially approved on the basis of income and household size. A full report presenting the results is forthcoming in Spring 2004.

Measuring Competitive Foods in Schools – Methods and Challenges
There has been growing concern about the sale in schools of foods with limited nutritional value. Of particular concern is the availability of “competitive foods,” a term that includes a wide range of foods that do not qualify as reimbursable meals under the National School Lunch Program or School Breakfast Program. Such foods can often be obtained from a la carte cafeteria sales, vending machines, and school stores. While the widespread availability of competitive foods is well documented, there is relatively little detailed data on the amounts of various types of competitive foods that are sold in schools or about their nutrient content.

FNS sought to develop a methodological basis for obtaining detailed information on the competitive foods sold in schools. Two research firms were contracted to develop and test two different methodologies for collecting these data. One methodology used an inventory approach to collect data on sales of competitive foods. The second methodology focused on directly observing the purchases of competitive foods at their “points of sale,” such as cafeteria checkout lines or vending machines. Each approach was tested in three schools and the data received were coded into a nutrient database and converted into information on nutrient content. The results of this methodological research are expected in Spring 2004.