Seal America, The Prevention Invention
Seal America, The Prevention Invention

Step 10
Program Evaluation

Applying high-quality dental sealants to as many students as cost-effectively as possible is every school-based dental sealant program’s goal. To determine whether a program is operating effectively and efficiently requires evaluating the program on a regular basis. Although a rigorous scientific evaluation is beyond the scope of most school-based dental sealant programs, there are several activities that program administrators can undertake to gain insights that can help improve program performance.

Because the effectiveness of dental sealants has been well documented.1, 2, 3, 4, 5 School-based dental sealant program administrators typically concentrate evaluation efforts on determining the quality of the dental sealants placed, calculating the program’s cost, and analyzing program participation.

Determining Dental Sealant Quality

A key indicator for measuring dental-sealant quality is retention rate. Sealant quality can be measured by checking short-term retention rates, 1-year retention rates, or both. Both measures serve important purposes.

Short-Term Retention Checks

A sample of students who receive dental sealants as part of a school-based dental sealant program can be evaluated a few days or weeks after sealant application to ensure that the dental sealants are intact, adequately cover the occlusal pits and fissures, and have marginal integrity. The proportion of students checked and the frequency with which they are checked varies from program to program. Conducting short-term retention checks can be an effective way to evaluate staff performance, identify needed protocol changes, and determine the adequacy of material and equipment used. Short-term retention checks offer an opportunity to correct problems with sealant-application techniques, material, and equipment. This type of retention checks can be especially useful in evaluating the performance of a new operator.

Yearly Retention Checks

Yearly retention checks can begin in the second year of the program and should occur yearly thereafter for as many students as possible. One-year retention rates of properly applied sealants should be high, averaging between 80 percent and 90 percent.2,6

For programs that provide dental sealants for second- and sixth-grade students, retention checks typically involve checking third-grade students who received dental sealants the year before. Upon graduation from sixth grade, many students change schools, making retention checks difficult. If good tracking systems are in place, after the fourth year of a program sixth-grade students can also be checked, thus documenting 4-year retention rates of dental sealants that were placed when these students were in second grade.

To perform yearly retention checks, advance preparation is required. First, student records from the previous year need to be retrieved from storage. Using third-grade class lists, students who received dental sealants the previous year can be identified. New room numbers should be marked on student records before the program is next scheduled to operate in the school, and the records should be taken to the school when the program operates. If electronic records are being used, a list of third-grade students with room numbers will need to be generated.

As resources allow, retention checks should be completed on as many students as possible. A space should be available on the student record to indicate retention check results.

Lost vs. Partially Lost Dental Sealants

Some programs differentiate between lost and partially lost dental sealants; others do not, because both lost and partially lost sealants result from moisture contamination during application process. Program planners need to establish retention-check criteria that clearly define lost and partially lost dental sealants so both are recorded properly. Partially sealed teeth are also identified during retention checks. (Teeth may be partially sealed, for example, when the sealant was applied before the occlusal surface of the tooth was fully erupted.) In these instances, an add-on sealant is indicated to protect the tooth. This tooth, however, should not be counted as a sealant because it was not completely sealed in the first place.

Program administrators must decide whether lost sealants will be reapplied. Some consent forms allow for reapplication of lost sealants that are detected during retention checks in subsequent years. In these instances, lost sealants are re-applied, and new sealants are applied to teeth that have erupted, if appropriate, within the past year. An attorney familiar with consent can help determine how consent forms should be phrased to cover retention checks and additional dental sealant placement.

After all the retention-check information has been collected, the retention rate should be computed for each student and the total number of students in each school. The process of conducting retention checks is similar to the process of conducting student assessments.

Determining Program Cost

The cost of a school-based dental sealant program can be measured in terms of cost per tooth sealed or cost per participating student. Funding sources often reimburse programs based on the number of students participating or on the number of teeth sealed. Before most funders decide whether to continue supporting the program, they will want to know these costs. School-based dental sealant programs must also be able to demonstrate that they are applying dental sealants as cost-effectively as possible.

To determine the cost of operating a school-based dental sealant program, records must be kept of all costs related to the program, including costs for staff, supplies, printing, equipment, equipment maintenance, and transportation. The following table reports costs so that program administrators can compare line-item costs, cost per student assessed, cost per student who received dental sealants, cost per tooth sealed, and cost per visit.

Table 10.1. Cost per Year of Operating a School-Based Dental Sealant Program

ITEM
Sealant Program Agency
Grants *
Donated **
Total
Staff (including benefits):        
Dentist(s)        
Dental hygienist(s)        
Dental assistant(s)        
Program administrator(s)        
Administrative staff (e.g., data-entry clerk, secretary)        
Supplies        
Printing        
Travel (e.g., transporting equipment, distributing consent forms, meeting with funders)        
Liability insurance        
Equipment (depreciate by dividing initial cost by life expectancy)        
Other (e.g., phone, postage, pager)        
TOTALS        
* Financial support from organizations other than the one operating the program.
**The value of goods and services provided at no cost from another organization.

 

Program Participation

Monitoring participation is important for determining the extent to which a program is serving its target population. Program participation can be measured in several ways.

If the total number of consent forms returned is low compared to the total number of students eligible for the program, program administrators can begin to gain insights into the perceived value of the program; the effectiveness of the program’s marketing to principals, teachers, parents, and students; and the clarity of the language used in the consent form. For example, a low percentage of returned forms might indicate a lack of commitment by teachers in collecting consent forms, or it might indicate that the forms are written at an inappropriate reading level.

Each returned consent form either grants (positive) or declines (negative) permission to apply dental sealants. After comparing the number of students assessed to the total number of positive consent forms, program administrators may want to re-examine their policies and practices that affect the rapport of program staff with school personnel, the effectiveness with which schools are scheduled, and the manner in which the program operates within schools. If insufficient attention is given to scheduled school activities such as special programs and field trips, some students who return positive consent forms may not be available for dental sealant application. Variables that cannot be controlled, such as bad weather, school closings, and outbreaks of illness in schools, also affect participation.

Program administrators can use participation information to determine whether the method for targeting students for the program is appropriate. If there is a large discrepancy between the number of students assessed and the number who receive dental sealants, it is possible that the teeth of students targeted for the program are not sufficiently erupted. This problem can be solved by targeting the program to students in a different grade. Based on an analysis of data from Ohio school-based dental sealant programs, the following formula was developed:

Table 10.2. Formula for Estimating the Percentage of Eligible Student with Positive Consent Forms That Receive Dental Sealants


55% of students eligible returned a positive consent form (e.g., of 100 students eligible 55 will return a positive consent form)
  92% of those students were assessed (e.g., of the 55 students who returned consent forms approximately 51 will be assessed)
    84% of those students needed dental sealants (e.g., of the 51 students assessed, approximately 43 will need dental sealants)
      95% of those students received dental sealants (e.g., of the 43 students needing sealants, approximately 41 will receive dental sealants)
        overall, 41% of students eligible received dental sealants


Program participation rates provide additional information if they are analyzed by age or grade and if they are compared to the participation rates of other school-based dental sealant programs with similar targeting criteria. Significant differences in participation rates among schools or changes in participation rates within a school from one year to the next deserve further investigation.

Information collected on the student record is useful to determine the oral disease experience of students participating in the program and their sealant needs, as well as how well the program is meeting those needs. For example, see Cincinnati Ohio’s student record.

The following table can help school-based dental sealant program administrators with evaluation. When making comparisons among programs, the method of targeting used should be considered. Most programs initially target either specific schools or individuals that meet criteria, which usually relate to age, income, disease experience, or the availability of oral health services.

By monitoring program participation in the ways discussed above, a program administrator can evaluate the impact of a school-based dental sealant program in a community, Sharing this information among programs can provide program administrators with a frame of reference not previously available.

Table 10.3. Dental Sealant Program

Participation and Assessment Information
TARGETING TAKES PLACE BY:
School
(Check all that apply)
Individual
(Check all that apply)
_____ Free or reduced-price meals program _____ Grade level
_____ Population-to-dentist ratio in community _____ Free or reduced-price meals program
_____ Federal poverty levels _____ Family income
_____ High caries rates (survey) _____ High caries activity
_____ No criteria _____ No family dentist
_____ Other _____ Other

Targeted Grades
         
Consent Number of:          
  a) Students eligible ***          
  b) Given consent**          
  c) Refused consent*          
  d) Unreturned consents          
(a = b + c + d)          
Assessment Number of:          
  a) Students assessed          
  b) Students with history of dental caries          
  c) Students with untreated caries          
  d) Students with one or more sealants          
Sealants Number of:          
  a) Students needing sealants          
  b) Students receiving sealants          
c) Teeth sealed          
Key:  
  *** Students eligible refers to students who receive consent forms.
** Positive consent refers to students eligible whose parents granted permission to participate.
* Negative consent refers to students eligible whose parents denied permission to participate.
 

 

Other Evaluation Tools

The New York State Department of Health, Bureau of Dental Health, has developed a performance effectiveness review self-assessment that itemizes indicators such as

  • Data and records management
  • Quality assurance
  • Infection control
  • Exposure control
  • Relationship with the community
  • Staffing
  • Equipment

CDC’s SEALS resource, which is discussed in Step 6, can also serve as a program-management tool. The tool allows for analysis of assessment data, comparison of events by student risk status, sealant prevalence, and participation rates. The tool also makes it possible to compare the cost of one school-based dental sealant program to another. Program administrators can also use the SEALS software to generate reports or to provide data to schools officials, policymakers, and funders. For more information about the SEALS program management tool, contact Susan Griffin at CDC.

References

  1. National Institutes of Health. 1984. Consensus development conference statement on dental sealants in the prevention of tooth decay. National Institutes of Health. Journal of the American Dental Association 108(2):233–236. http://consensus.nih.gov/1983/1983DentalSealants040html.htm.
  2. Ahovuo-Saltoranta A, Hiiri A, Norblad A, Worthington H, Makela M. 2004. Pit and fissure sealants for preventing dental decay in the permanent teeth of children and adolescents. Cochrane Database of Systematic Reviews (3):CD001830. http://www.ncbi.nlm.nih.gov/pubmed/18843625.
  3. Rozier RG. 2001. Effectiveness of methods used by dental professionals for the primary prevention of dental caries. Journal of Dental Education 65(10):1063–1072. http://www.ncbi.nlm.nih.gov/pubmed/11699978.
  4. Truman BI, Gooch BF, Sulemana I, Gift HC, Horowitz AM, Evans CA, Griffin SO, Carande-Kulis VG, and Task Force on Community Preventive Services. 2002. Reviews of evidence on interventions to prevent dental caries, oral and pharyngeal cancers, and sports-related craniofacial injuries. American Journal of Preventive Medicine 23(1 Suppl):21–54. http://www.ncbi.nlm.nih.gov/pubmed/12091093.
  5. Weintraub JA. 2001. Pit and fissure sealants in high-caries-risk individuals. Journal of Dental Education 65(10):1084–1090. http://www.ncbi.nlm.nih.gov/pubmed/11699981.
  6. Dorantes C, Childers NK, Makhija SK, Elliott R, Chafin T, Dasanyake AP. 2005. Assessment of retention rates and clinical benefits of a community sealant program. Pediatric Dentistry 27(3):212–216. http://www.ncbi.nlm.nih.gov/pubmed/16173225.

 

 

Home Acknowledgments FAQ's