PBIS Evaluation For EdTech 505
PBIS Evaluation For EdTech 505
Evaluated by: Jennifer Harris For: EdTech 505 Evaluation of Educational Technology Fall 2012
2 Summary The purpose of the PBIS program is to encourage school wide positive behaviors and choices in the student population served. All staff members are expected to participate in this program. The program includes a system of rewards for positive behaviors and a way to document behavior issues. Teachers also have a defined set of major and minor behavior types which include documentation forms. While the consequences of behavior issues vary by classroom, all referrals are entered into the HEART system. This system allows users to look at various types of behaviors, identify behavior trends and provides teachers, PBIS team members, and interventionists, with names of students who might benefit from behavior interventions and behavior intervention plans (BIPs). The PBIS team has a plan in place for making informed decisions based on this data. The major goal of the PBIS program is decrease the number of negative behaviors accrued by students. The purpose of this evaluation is to identify the use and impact of the PBIS referral and intervention system used by the faculty at Thomas Jefferson Elementary. The evaluation explores both the use of the behavior system and the impact it has had on the negative behaviors of current third graders in the school. The participants will be staff who have been part of the district for 2009-2012 school years, and 3rd grade students who attended TJ from December 31 , 2009 through October 15th, 2012. School staff members have been surveyed to determine the levels of participation in PBIS and how this may affect behaviors and evaluation data. Data concerning the behavior referrals has been collected to determine the impact the system has had on the students. Data from the last three years has already been compiled in a behavior data system, called the HEART system, and data from the current 2012-2013 school year is continually entered into the system as it is compiled. The first major evaluation task was to determine if any school staff were not district employees during the 2009-2012 school years, then to survey those that meet that criteria. Also, data for students who do not fit the attendance criteria of the evaluation have been removed as well. Staff members were surveyed to document their participation in the program and data from the behavior system was compiled, noting the number of referrals written by each teacher. Next, students were coded according to total numbers of referrals in each school year of attendance, and students with 6+ referrals in a given year were analyzed for behavior interventions and plans. The evaluation found that about half of the survey participants use PBIS interventions and referral forms as the behavior matrix indicates. About half of the participants indicated that they sometimes, rarely, or never use minor or major referral forms. Staff also indicated that they do not use the data found in the referral reporting system to discuss and prioritize students with team members. Despite these trends amongst staff, students who are considered habitual offenders, those with 6 or more referrals in a given year, have displayed a decrease in problem behaviors between their kindergarten and 3rd grade years of school at Thomas Jefferson Elementary.
st
Description of the Program Evaluated The PBIS program at Thomas Jefferson Elementary (TJ) was intended to help encourage positive behaviors and provide a way to document the inappropriate behaviors of the students in attendance. All staff members at TJ are expected to use the program as a means for improving school climate and providing all classes with a set of school wide expectations. TJs school wide expectations are for all students to be respectful, responsible, and safe. Additionally, expectations
3 for the commons areas (lunchroom, playground, etc.) are taught by following a set of school wide lesson plans provided to staff at the beginning of the year. All classes are assigned different times in the first two weeks of school where selected teachers will model expected and acceptable behaviors. The selected teachers are those who have a duty in that area and will be ultimately responsible for following through with consequences. These plans are revised as needed by the PBIS team each year. Individual lessons are re-taught during the year based on data collected in the HEART system. The intent is for any staff member to be able to walk down the hall, into the lunchroom, on the playground, or into any other classroom in the school and reward these expectations for being followed. PBIS also provides staff with a way to address appropriate and inappropriate expectations in a common way. Behavior problems are addressed using minor and major referral forms (Appendix B & C), while red pride and gold pride tickets are used to reinforce positive behaviors. Examples of behaviors which fall under these referral types can be found in the behavior matrix (Appendix A). Data from referrals are used to make informed decisions on lesson plans, behavior plans, and individual behavior interventions. Other than pre-school opportunities, Thomas Jefferson is the first school experience for many of the students it serves. Students are continually learning the behavior expectations necessary for appropriate school interactions. In this case, PBIS does not have a termination time or date, because the school will continually experience an influx of new students and an intent for promoted students to show growth in behavior choices. Teachers are expected to follow through with PBIS expectations and guidelines, including the use of pride tickets and referrals throughout the school year. The PBIS teams goal is 80% participation by the end of the 2013 school year. There are three steps in identifying and writing forms for inappropriate behaviors, minor, intervention, and major referrals. Please see Appendix A for the behavior referral matrix used by staff. Program Objectives This evaluation looks to determine if the following objectives have been met regarding the use of the behavior documentation and intervention system. The objectives were determined through conversations with the PBIS facilitator and research from previous meetings. Objective 1: A minimum of 80% of staff members will use the PBIS behavior documentation system. Objective 2: Staff will use behavior interventions and documentation according to the staff handbook. Objective 3: Staff will use data to monitor and discuss individual student behaviors and groups of students. Objective 4: Staff will use minor and major referral forms to communicate with parents. Objective 5: The use of referral forms, interventions, and BIPs will decrease the number of negative behaviors in students. Program Components One important component of the PBIS program is the ways which negative behaviors are addressed. Staff members are expected to fill out a tri carbon packet called a referral for negative behaviors (Appendix B&C). One sheet stays with the homeroom teacher, a second goes home
4 for parents to sign, and a 3rd goes to the guidance counselor or school secretary for HEART system filing. Minor referrals are signed by parents and returned which are also entered into a data system by the school guidance counselor. Major referrals are sent to the school principal who then addresses the problem with the student and parents. The secretary is responsible for entering this data in the behavior database. All data is entered into the HEART system (Appendix I), an online computer based system, provided by the Area Education Association (AEA). This data is used to make decisions concerning re-teaching, as well as in the meetings noted below. A second component is the use of a behavior intervention team. Students with and without BIPs are subject to these interventionists when behaviors fit in specific areas of the behavior matrix (Appendix A). When a student is exhibiting behaviors which cause a major disruption to student learning or behaviors which may harm the student or others, a designated behavior interventionist is called to the area. This interventionist is trained in defusing situations, de-escalation, and, if necessary, restraining of the child. The interventionist often removes the child from the situation and into a non-threatening area. These interventionists, and staff members, administration, and parents are also involved in the fourth component of PBIS which is the identification of students for Behavior Intervention Plans (BIP). BIPs include a variety of individual interventions aimed at improving identified student behaviors. Sometimes the BIP mandates that the interventionist be called in the event of any inappropriate behaviors. A final component of the PBIS program at TJ is the use of a positive rewards system. The positive rewards used by staff at Thomas Jefferson Elementary include red and gold pride tickets, all school assembly recognition, weekly academic award winners, class awards, special lunch seating, and more. Though it is an important part of the program, the focus of this evaluation is not on the positive rewards aspect of PBIS. Evaluation Method Participants In 2010, the Newton Community School District restructured. All Kindergarten, 1 , 2 , and 3 graders are now in two buildings, Thomas Jefferson Elementary (TJ) and Berg Elementary (BE), as opposed to five K-6 buildings. This resulted in TJ and BE restructuring their PBIS programs to tailor to the needs of younger students. Both schools have used these structures for three years and as part of the current school year, indicating that now might be a good time to evaluate the effectiveness and impact on the teachers who use the program and the students served by the program. Staff members are participants in the evaluation process. As mentioned above, staff members participate in the use of PBIS to varying degrees. Part of determining the impact of PBIS will be determining to what level staff members participated. Staff members have been provided with a survey concerning their use of the PBIS referral and intervention system (Appendix F). Data is available on the HEART system which breaks down the number of referrals written by individual teachers according to year (Appendix D). Teachers, classified staff (para-professionals, secretaries, AEA personnel), and administrators are responsible for writing out the referrals which are the data of the HEART system. Para-professional use of the PBIS system was not evaluated because only three of the sixteen para-professional staff members have been part of the TJ team for more than 1 year. In addition to general staff, several instructors and administrators play an even larger role in the PBIS program. The PBIS team is made up of the school principal, guidance counselor,
st nd
rd
5 special education team, and a representative from each grade level or team. The guidance counselor and principal facilitate semi-annual meetings and keep records of communications and issues. The principal and special education team also take on the role of the behavior intervention team discussed above. Procedures Staff members received initial training on the use of referrals and interventions, and subsequent trainings as needed thereafter. At the beginning of each school year, the PBIS team reviews the behavior matrix and lesson plans for the common areas with the teaching staff. At most of the monthly staff meetings, the guidance counselor reports out on behavior trends, identifying areas that require re-teaching. The PBIS committee members initially met on a monthly basis to discuss areas of concern and change. As time progressed, the team began to meet on an as needed basis. The evaluation procedures included: 1. Determine which staff fit seniority criteria for survey (in district from 2009-2012, etc.) 2. Survey staff to identify trends in the use of the referral and intervention system and the use of data. (Appendix F) 3. Use the HEART system to determine staff use of written forms. (Appendix D) 4. Determine which students fit the attendance criteria according to Infinite Campus (Appendix I): 1. Attended NCSD from 2009-2010 (Kindergarten) 2. Attended Thomas Jefferson from December 31 , 2009-October 15 , 2012 3. Did not attend another school during that timeframe 5. Identify students, according to the HEART system, who have: (Appendix J) 1. 0 referrals in the time frame provided 2. 2>referrals during their Kindergarten year and 0 in following years 3. 6>referrals in any given year 4. 6+ referrals in a given year
st th
1. Identify Students who have had a Behavior Intervention Plan (BIP) or have had intervention or other services related to behavior 2. Chart trends in inappropriate behaviors in students with 6+ referrals in any given year. (Appendix K) Data Sources PBIS Behaviors Survey: APPENDIX F, provide data on staff use of the referral and intervention system. This survey was developed by the evaluator and the PBIS team leader. Infinite Campus: APPENDIX I and APPENDIX J detail where students attended school in grades K-3 . This system was used to eliminate students who do not fit the evaluation criteria such as those students who did not attend school in the district during the 2009-2010 school year, and students who were retained for an extra year at TJ.
rd
HEART System: APPENDIX D and APPENDIX I, HEART system screenshots, provide detailed data concerning:
6 1. Number of referrals written by teacher, grade, and team each year 2. Total referrals in any year by grade 3. Year (dates) of each referral occurrence for each student Results Staff Survey 20 of 35 staff members responded, or 57% of staff Statement: USE I use minor referral forms for inappropriate behaviors I use major referral forms for inappropriate behaviors. I use minor and major referral forms according to the behavior matrix found in the staff handbook. INTERVENTION I call an interventionist when a child is a disruption or causing a safety issue. I call an interventionist prior to writing a major referral. I call an interventionist when behaviors fit the criteria found on the behavior matrix located in the staff handbook. DATA I use minor/major referral data to discuss students behaviors with my team. I use minor/major referral data to discuss students behaviors with interventionists. I use minor/major referral data to discuss students behaviors with appropriate individuals outside of my team (guidance counselor, specials teachers, etc.) I contact PBIS team members to look at data on problem students for prior years. My team uses referral data to prioritize students before suggesting behavior intervention meetings and plans. Communication I call parents when a child receives a minor referral form. I use another means to contact parents in addition to sending home referral sheets. I follow up with the return of parent signed referral forms. I implement consequences after writing minor referrals. I re-teach the lesson plans for expectations when the data shows it is necessary. I re-teach the lesson plans for expectations for the common areas when the data shows it is necessary. Attitudes I feel PBIS referrals and interventions have a positive effect on decreasing the number of problem behaviors. I feel PBIS referrals and interventions are an effective way to document problem behaviors Responded Often or Almost Always 42% 47% 89% 74% 41% 58% 37% 53% 63% 16% 32% 16% 42% 68% 89% 68% 68% 68% 74% Responded Sometimes, Rarely, or Never 58% 53% 11% 26% 59% 42% 63% 47% 37% 84% 68% 84% 58% 32% 11% 32% 32% 32% 26%
HEART System Grade/Team Referral Data Staff % Referrals Written 2010-2011 % Referrals Written 2011-2012 School Year School Year Kindergarten 22% 27% st 1 Grade 42% 13% nd 2 Grade 7% 9% 3rd Grade 4% 6% Auxilary Staff 5% 11% (Specials, Title, Reading, etc.) Intervention/SpEd 20% 34% 356 Total Referral 399 Total Referrals View extensive data broken down by teacher, team/grade level, and year in APPENDIX E Infinite Campus - Student Numbers 3rd Grade Population Total (As of Oct. 15, 2012) 100 Students Did Not Attend District 2009-2010 or TJ from 2010Oct. 15, 2012 16 Students Not promoted to next grade between 2010-2012 2 Students Attended District 2009-2010 and TJ from 2010-Oct. 15, 2012 82 Students
The data of 82 students was used to determine habitual offenders, or those students with 6+ referrals in a given year in the following graphs. School Year Total Heart Data System *Total Referrals for the 82 Referrals* Students who meet attendance criteria above 2009-2010 193 (right side graph) 2010-2011 169 2011-2012 98 Aug. 2012-Oct. 15, 2012 12 HEART Data System Student Data 39% of the students included in data mining have never received a referral, and another 9% received less than 2 referrals their kindergarten year then never received another in subsequent years. However, 22% of students had 6+ referrals between 2009 and October 15th, 2012. (Appendix J) HEART Data System Habitual Offender Trends (students with 6+ referrals in any year) The graphs on the
8 left (APPENDIX K) depict the referral trends of students who had more than 6 referrals in any year from 2009-October 15th, 2012. Though not as significant, data for the 2012-2013 school year was included to show a change in behavior. Each graph shows a peak in referral numbers in either the Kindergarten or first grade year for all students and decrease in the student receiving referrals in subsequent years. Students A and B, who show no referrals in the 2011-2012 school year, but referrals in the 2012-2013 school year, both had teachers who did use the referral documentation system during that time period. Student % Decrease in Referrals (K+1st)-(2nd+3rd)/Total A 97% B 66% C 79% D 45% E 81% F 63% G 84% H 74% I 55% J 19% K 47% L 78% M 71% N 33% Students without interventions or behavior plans averaged an 81% decrease in referrals between their kindergarten-first grade years and second-current year. Students with a behavior plan averaged a 77% decrease between K-1 and 2-3 years, while students who had interventions without a BIP only experienced a 54% decrease in behaviors during those same timeframes. Discussion The purpose of this evaluation is to determine if staff members are using the PBIS referral documentation system, and if they are using it in a way that impacted the amount of negative behaviors in students. A staff survey and referral data were used to evaluate the former, while the referral data of current 3rd grade students was used to determine the latter. Objective 1: 80% of staff members will use the PBIS behavior documentation system. Objective 4: Staff will use minor and major referral forms to communicate with parents.
9 Partially Met APPENDIX E, G, & H 79% of teaching staff surveyed used the PBIS behavior documentation system in the 2010-2011 school year. An increase of 9% occurred in the usage of the behavior documentation system in the 2011-2012 school year, resulting in 88% usage. The PBIS teams 3 year goal of 80% usage has been met by the teaching staff. The para-professional staff will need to evaluated in subsequent years when enough data has been collected in the HEART system to confirm 80% usage by all staff. (Only 3 of 16 para-professionals have been staff members for more than one year.) Despite meeting the 80% mark for usage, the staff survey shows that more than half of teachers who participated admit that their usage of minor and major referral slips fall under the category of sometimes, rarely, or never. This could explain why data shows a high usage among kindergarten and first grade teachers and the lower usage among second, third, and auxiliary teachers. Recommendation: Meet with 2nd, 3rd, and auxiliary teams to determine why they are not writing as many referrals. Especially address the areas of student population, student maturity, parental involvement, and the referral documentation method. Objective 2: Staff will use behavior interventions and documentation according to the staff handbook. - Not Met APPENDIX F, G, & H There are several areas addressed in the handbook which apply to PBIS compliance in staff. More than half of participants say that they sometimes, rarely, or never use the minor and major referral forms, or call for an interventionist before writing a major referral. 89% of survey participants indicated that they use the behavior matrix found in the staff handbook to determine the severity and level of behaviors, but only 58% of participants say that they call for an interventionist when behaviors fit that criteria. These areas are important to the programs success and should therefore be monitored for their efficiency and effectiveness. Recommendation: Use the HEART data system information to meet with individual teachers who have not adequately used the documentation system. Reinforce the idea of a joint team effort in promoting appropriate behaviors. Re-teach the behavior matrix to staff and use the matrix to make clear to staff that they should call for intervention, even in cases where the behavior is a first time problem such as profanity, lying, or theft. Use staff conversations to determine why teachers do not call for intervention when severe behaviors fall under intervention in the matrix. Objective 3: Staff will use data to monitor and discuss individual student behaviors, and groups of students. - Not Met APPENDIX F Only 16% of participants contact PBIS team members to look at data regarding problem students. Less than 40% of survey participants use referral documentation data to discuss problem students with grade level team or use data to prioritize students for behavior interventions and plans. While higher than those numbers, only about half of staff members surveyed use referral data to discuss problem students with interventionists and other relevant teachers (specials, guidance, etc.). 68% of staff surveyed say that they use data to inform their decisions regarding the re-teaching of building and commons area expectations for large groups of students. This type of information is often presented by the guidance counselor at staff meetings.
10
Recommendation: Provide staff with adequate access to data system. Either provide staff with their own username and password, or create a staff login. Provide training on the use of HEART system and ways which data can be viewed and calculated (by student, month, behavior, etc.). Objective 5: The use of referral forms, interventions, and BIPs will help decrease the number of negative behaviors in students. Met APPENDIX J & K The most staggering statistics is that students with no interventions or behavior plans averaged an 81% decrease in referrals between their kindergarten and first grade years and second and current year. Possible reasons for this decline could include parental intervention as well as that, according to the staff survey, 89% of participants initiate consequences related to the referrals given to students. 68% of staff survey participants say that they also follow up with the signing of referrals by parents, resulting in better teacher-parent-student communication. Habitual offenders with a behavior plan averaged a 77% decrease between K-1 and 2-3 years, while the largest group of students, those who had interventions without a BIP, only experienced a 54% decrease in behaviors during those same timeframes. In general, there was a 42% decrease in referrals between the 82 3rd grader participants 1st and 2nd grade years. If this group of children maintains its current behavior referral trend, they should receive about 60 referrals during the entire 2012-2013 school year, a decrease of 39% between 2nd and 3rd grade, and a decrease of 64% between 1st and 3rd grades. Recommendations: Increase current use of referral documentation and interventions in staff members. Identify more students who are habitual offenders and consider BIPs for those students. Increase parent communication follow ups. Proposed Budget Personnel: Evaluator: 25 days at $500/day Travel and per Diem: 70 mi/day for 4 days at $.55/mi Miscellaneous mileage: 250 miles at $.55/mi Communications: Telephone and Internet ($120/month for 1 month) Supplies, Materials, and Photocopying Photocopying Supplies and Materials $12,500 $12,500 $154 $137.50 $291.60 $120 $120 $12,500
$291.50 $120
$200 $100 $300 $300 TOTAL BUDGET $13,211.50 The evaluator will need to travel to the evaluation site at least three times during the evaluation to meet with stakeholders, and once to present. Miscellaneous mileage includes extra trips to the evaluation site, mileage for photocopying, purchasing supplies, and delivering the final evaluation report to the staff. Internet access is especially important given that the HEART
11 data system is fully online and the staff survey will also be presented online. Several copies of the evaluation reported will be provided to the PBIS team and staff members.
17
18
19 Staff Kindergarten 1st Grade 2nd Grade 3rd Grade Auxilary Staff (Specials, Title, Reading, etc.) Intervention/SpEd % Referrals Written 20102011 School Year 22% 42% 7% 4% 5% 20% 356 Total Referral % Referrals Written 2011-2012 School Year 27% 13% 9% 6% 11% 34% 399 Total Referrals
DATA I use minor/major referral data to discuss students behaviors with my team. I use minor/major referral data to discuss students behaviors with interventionists. I use minor/major referral data to discuss students behaviors with parent(s). I use minor/major referral data to discuss students behaviors with appropriate individuals outside of my team (guidance counselor, specials teachers, etc) I contact PBIS team members to look at data on problem students for prior years.
21 Almost Always
DATA My team uses referral data to prioritize students before suggesting behavior intervention meetings and plans.
COMMUNICATION I call parents when a child receives a minor referral form. I use another means to contact parents in addition to sending home referral sheets. I follow up with the return of parent signed referral forms. I implement consequences after writing minor referrals.
Almost Always
RE-TEACHING I re-teach the lesson plans for expectations when the data shows it is necessary. I re-teach the lesson plans for expectations for the commons areas the data shows it is necessary.
Almost Always
ATTITUDES I feel PBIS referrals and interventions have a positive effect on decreasing the number of problem behaviors. I feel PBIS referrals and interventions are an effective way to document problem behaviors.
Almost Always
18.
19. 20.
A: ALMOST ALWAYS QUESTION Participant A Participant B Participant C Participant D Participant E Participant F Participant G Participant H Participant I Participant J Participant K Participant L Participant M Participant N Participant O Participant P Participant Q Participant R Participant S
1
N:NEVER
16 17
S O R S O S R S A A S A O S O O S S S
2 A O S A S S O S A A R A O A S R S R S
S O R O S A R O A A A O R A S A O O A
S O R O S O R O A A A O R A S A O O O
O O R O O O S A A S A O O A S A S S O
18 S O O S A O S A A S A O O A S A O O A
19 O O S O A A S O A A A O A O A A O A O
20 S A R A A S A O A S S A S O R A O A S
23 APPENDIX H: STAFF SURVEY RESPONSES: OFTEN/ALMOST ALWAYS & SOMETIMES/RARELY/NEVER Staff Survey 20 of 35 staff members responded, or 57% of staff Responded Responded Often or Sometimes, Statement: Almost Rarely, or Always Never USE I use minor referral forms for inappropriate behaviors 42% 58% I use major referral forms for inappropriate behaviors. 47% 53% I use minor and major referral forms according to the behavior matrix 89% 11% found in the staff handbook. INTERVENTION I call an interventionist when a child is a disruption or causing a 74% 26% safety issue. I call an interventionist prior to writing a major referral. 41% 59% I call an interventionist when behaviors fit the criteria found on the 58% 42% behavior matrix located in the staff handbook. DATA I use minor/major referral data to discuss students behaviors with 37% 63% my team. I use minor/major referral data to discuss students behaviors with 53% 47% interventionists. I use minor/major referral data to discuss students behaviors with 63% 37% appropriate individuals outside of my team (guidance counselor, specials teachers, etc.) I contact PBIS team members to look at data on problem students for 16% 84% prior years. My team uses referral data to prioritize students before suggesting 32% 68% behavior intervention meetings and plans. Communication I call parents when a child receives a minor referral form. 16% 84% I use another means to contact parents in addition to sending home 42% 58% referral sheets. I follow up with the return of parent signed referral forms. 68% 32% I implement consequences after writing minor referrals. 89% 11% I re-teach the lesson plans for expectations when the data shows it is 68% 32% necessary. I re-teach the lesson plans for expectations for the common areas 68% 32% when the data shows it is necessary. Attitudes I feel PBIS referrals and interventions have a positive effect on 68% 32% decreasing the number of problem behaviors. I feel PBIS referrals and interventions are an effective way to 74% 26% document problem behaviors
25 APPENDIX J: Student Data Infinite Campus - Student Numbers 3rd Grade Population Total (As of Oct. 15, 2012) 100 Students Did Not Attend District 2009-2010 or TJ from 2010Oct. 15, 2012 16 Students Not promoted to next grade between 2010-2012 2 Students Attended District 2009-2010 and TJ from 2010-Oct. 15, 2012 82 Students
Total Referrals For 82 Students by Year School Year Total Referrals* 2009-2010 193 2010-2011 169 2011-2012 98 Aug. 2012-Oct. 15, 2012 12 Heart Data System *Total Referrals for the 82 Students who meet attendance criteria above
27
Student % Decrease in Referrals (K+1st)-(2nd+3rd)/Total A 97% B 66% C 79% D 45% E 81% F 63% G 84% H 74% I 55% J 19% K 47% L 78% M 71% N 33%
28