Evaluation Tool
Site Visit Protocols for Conducting Program Reviews
of Title X Family Planning Projects
Purpose of This Document
This document is a revision to the 1990 site review protocol developed
jointly by the ten regional family planning offices of the U.S. Public Heath
Service (PHS). The document serves as a guide for the periodic review of
Title X Family Planning grantees and delegates. It contains questions to
structure on-site inquiry into four primary program areas: administration,
finance, clinical counseling and educational services, and community
outreach and information. The revisions include additional questions
regarding grantee/delegate agency utilization of technical assistance and
training, male, adolescent and other high risk population initiatives,
electronic communication, new contraceptive method management, and community
partnering.
This protocol has a dual function. First, it serves as a basic tool for
determining, at a minimum, whether Title X grantees and delegates are
complying with the provisions of Title X of the Public Health Service Act of
1970 and subsequent amendments. It provides a checklist for gathering
quantifiable data and allows for suggesting benchmarks for corrective action
within a given time frame. Secondly, use of this protocol by regional staff
and grantees will establish priorities and document long term plans for
program assistance which may include consultation, technical support, or
specific training activities.
Consistent utilization of this document for site visit reviews nationally
will support and promote a uniformly high standard of quality in family
planning services without compromising the flexibility in services offering
necessary to accommodate national, regional or local needs.
Return to top
Philosophy and Use of Program Review
The program review is a valuable management tool that provides planning
information to Federal regional offices and family planning grantees. The
program review is not an independent activity, but rather exists on a
continuum of services performed by the regional offices in their stewardship
of the U.S. Public Health Service (PHS) Title X Family Planning Program. The
Federal government requires that all family planning projects funded under
Title X be monitored to ensure their compliance with provision of the
“law”. This monitoring, which is done primarily through periodic on-site
reviews, is one of the several components that collectively constitutes an
overall quality assurance system.
Program review site visits generate data that form the basis for Title X
grant action decision and help regional offices take corrective action and
grantees identify program areas that require additional support to improve
program activities. Moreover, analysis of the data can produce information
about the needs of grantees and information necessary for developing and
carrying out responsive technical support, consultation, and training
opportunities.
Return to top
Program Review Objectives
The primary objectives for the PHS Title X program review are as follows:
- To ensure grantees’ understanding of and compliance with PHS Title X
and other federal laws, regulations, policies, and guidelines.
- To verify activities reported by the agency, such as the agency’s
expenditure of funds and scope of services.
- To determine each grantee’s progress in implementing the project plan
as stated in the approved Title X grant application.
- To determine whether appropriate systems are being implemented, are
effective and are ensuring program compliance with PHS Title X.
- To develop and present to the grantee a statement of findings and
recommendations including identified problem areas that require resolution
and, when appropriate, providing identified on-site consultation and
technical support.
- To obtain from the grantee an appropriate corrective action plan based
on the statement of findings and recommendations.
- To provide grantees an opportunity to develop a plan of assistance to
improve delivery of PHS Title X services by providing grantees resources
and follow up to questions or problems identified during the site
evaluation.
- To provide grantees the opportunity to identify trends and common
issues in the delivery of PHS Title X services and in the implementation
of Title X requirements to assist in federal program planning.
- To establish rapport with the agency grantees.
Return to top
Procedures for Conducting Site Visits and
Subsequent Follow Up
- Determine which grantee and, when appropriate, delegate(s) are to be
reviewed.
- Notify the grantee about the review in sufficient time; preferably 40
to 60 days prior to the evaluation visit.
- Work with the grantee to determine the dates of the visit and which
delegates are to be reviewed.
- Forty days prior to visit, send the grantee agency a letter indication
the following:
- Advance date requirements
- Site visit dates
- Visit schedule
-and-
- Entrance conference: who should attend, time, date, location
- Conference exit: who should attend, time, date, location
- Names of evaluation team members
- Specific instructions about:
- Setting aside a room for the evaluation team to use during the
site visit.
- Identify individuals who should be available, agency director,
governing or advisory board member(s), family planning coordinator,
medical director, fiscal manager, medical providers, other key family
planning program staff.
- Documents required.
- Scheduling for observation of all types of clinical service
delivery.
- Provide grantee with a copy of the program review tool.
- Determine the composition of the evaluation team, contact team members
and make assignment for responsibilities.
- Conduct entrance conference: Evaluation coordinator describes process,
introduces the evaluation team and reviews the agenda for site visit. The
agency and site visit schedule may be amended at the entrance conference,
when needed. The agency director should introduce the project staff,
individuals responsible for areas covered by the protocol and special
instructions for the evaluation process. A tour of the agency is conducted
after the entrance conference.
- Conduct the review and verify the findings. Evaluators conduct the
review by interviewing staff, observing client/staff interaction,
reviewing written materials.
- Conduct exit conference: Evaluation team members report finding and
recommendation. Agency staff is given opportunity to respond and ask
questions. Evaluation coordinator summarizes the meeting and advises
agency when final report will be completed and forwarded to the agency.
Training and technical support availability should be discussed at this
time, when appropriate. Agency is advised of follow up visit, if
necessary. Follow up visits occur usually within six months after the
evaluation report is received by the agency or on an as needed basis.
- Provide grantee with a report of findings and recommendation and a
timetable for submitting the corrective action plan. If possible, the
final evaluation report should be sent to the grantee within six weeks
following the completion of the site visit.
- Request grantee provide written response within six weeks after
receiving site visit report.
- When necessary, conduct follow visit to grantee and assess progress
made in implementing the corrective action plan.
Return to top
Program Review
Protocol
This review protocol has been developed to help provide consistency in
the evaluation process. The protocol is based on the requirements and
recommendations contained in the Federal Guidelines for Family Planning
Services.
The following legend will be used for all four review areas,
administration, finance, clinical services and community outreach and
information:
LEGEND
L -
“Law” Title X of the Public
Health Service Act of 1970, (PL. 91-572)
R - “Regulations” as published in the
Federal Register, 3 June 1980
G - Program “Guidelines” issued for
Family Planning Sources Project Grants
A - Standard for “Ambulatory” Health
Care as published by U.S. Public Health Services,1977.
At the beginning of each section, appropriate definitions are given along
with the goals and objectives specific to that particular program area. A
series of questions is listed on the left with column for checking results
to the right; “A” for an acceptable level of attainment of the criteria; “U”
for an unacceptable level of attainment of the criteria. The far right hand
margin of each page is reserved for the evaluator’s comments, which should
be useful when planning follow up program assistance.
New Mexico Family Planning Site Review Responsibility Coded as follows:
S - State FP Office
D - District Office
L - Local Health Office
C - Contractor
PA - Provider Agreements
Return to top
Adobe Acrobat Reader is required to read the
following sections.
If you do not have
the Reader, you may download it for free by clicking here -
Return to top
Community
Education
and Community Outreach
Community education in this document refers to going out into the
community and educating the public about health matters that concern them
and us as public health personnel. This may be accomplished, for example, by
a health educator going to the local high school and educating students
about STDs or contraception.
Community outreach in this document refers to recruiting clients in the
community to access our services. This may be accomplished, for example, by
advertising in the local paper, running Public Service Announcements on the
radio; distributing flyers at the local Laundromat, health fairs, etc., or
inviting people to visit our clinic for an exam after having given them an
educational presentation on STDs.
In order to comprehensively implement community education & community
outreach activities, it is helpful to have an overall plan. The following is
a suggestion on the development and implementation of such a plan.
An overview of planning for the community outreach and education might
initially (and perhaps optimally) be done at the district level. Please
refer to the illustration.
Planning Committee
Key players at an initial meeting might be as represented.
It is imperative, however, that the District Director
takes the lead and directs efforts.
The district level administrators would work with local health office
personnel in discerning who in the health office should be on their local
community outreach and education team. The district level administrators
could also, with their oversight, keep in mind the big picture for the
district and offer some consistency and continuity between the local health
offices regarding methodology. Furthermore, the district administrators
could be a resource for their health offices offering insight and
suggestions about what is working for various health offices. This
information could be very valuable for any health office encountering
barriers to their planning and implementation process.
At the local level, the planning may/should involve an initial stage of a
Community Needs Assessment: the development of a tool (or use of an existing
one), its distribution, collection, and analysis. This may be done in
cooperation with, for example, the county MCH councils.
Based on the Needs Assessment, goals & objectives would be developed and
activities implemented for community education & community outreach.
Possible actions for the local health offices to take might include the
following (which, for the Family Planning clinics, would fulfill Title X
Regs for community outreach & community education ( refer to Family Planning
Clinic Site Evaluation tool):
- Designate staff to develop plan;
- Develop a written plan for community outreach & community
education;
- Collaborate with key community contacts in developing and executing
the plan (both within and outside the DOH);
- Include in the written plan:
- who shall carry out plan activities;
- implementation of community needs assessment delineating mechanism(s)
to be used;
- results of needs assessment;
- goals and objectives responding to needs (include target
populations);
- planned activities to meet goals and objectives (e.g., educational
presentations, health fairs, media campaigns, etc.);
- sites to target (for example: schools, youth agencies, Promotora
project, PHD MOAs & Contractors
- needed community contacts (for example: principals, school nurses,
parents, executive directors, other non-PHD community educators, etc.)
- methods of client recruitment;
- community involvement opportunities;
- schedules/time tables
- Include in the plan, the ways and means for monitoring implementation
which might include:
- who will do the monitoring and what is the rational for their
participation in the monitoring process;
- what is the mechanism for monitoring the plan;
- how often will monitoring activities occur;
- evaluation criteria and procedures;
- any other relevant information.
- Document:
- carried out activities as they relate to the plan;
- the monitor’s evaluation of the success of plan implementation; if
any, monitor’s recommendations for activities or tasks to address any
deficiencies in the plan or its execution; compliance with
recommendations.
- Write up an annual report regarding the successes and obstacles
encountered.
Adobe Acrobat Reader is required to read the
following sections.
If you do not have
the Reader, you may download it for free by clicking here -
Task Checklist for Community Education &
Community Outreach
Community Outreach and
Information Section
Return to top |
NM Department of Health Family Planning Program
