College of Education Institutional Report

Standard 2. Assessment System and Unit Evaluation

The unit has an assessment system that collects and analyzes data on the applicant qualifications, the candidate and graduate performance, and unit operations to evaluate and improve the unit and its programs.

2.1 Assessment System and Unit Evaluation

How does the unit use its assessment system to improve candidate performance, program quality and unit operations?

In 2009, through a collaborative process that included faculty, program coordinators, school personnel, and other stakeholders, the School of Education (CEd) built a new assessment system to align our curricula with the newly adopted North Carolina 21st Century Professional Teaching Standards (I.5.c.6). Our system required that candidates produce six state-mandated electronic evidences that demonstrate depth and breadth of content knowledge, content pedagogical knowledge and skills, impact on P-12 student learning, and leadership. Advanced licensure candidates were required to produce two electronic evidences focused on leadership and assessment. Candidates in the Master of School Administration (MSA) program were required to produce electronic evidences that illustrate content knowledge, management, leadership, and reflective practice. Faculty reviewed and redesigned core professional education courses and specialty area courses. The Unit’s Professional Education Conceptual Framework (I.5.c.1) was also revised.

The assessment system is organized by four transition points at the initial and advanced licensure levels. At the initial level, these transition points are organized as four Professional Learning Communities (PLCs). At PLC I-Emergent Phase, education majors engage in foundational school-level observations where they focus on the moral purpose of education. At PLC II-Developing Phase, education majors in initial programs are formally admitted into the program.

Individuals admitted into the initial teacher education program enter PLC III- Developing/Proficient Phase where they engage in 40 hours of field experiences in P-12 school settings focused on differentiated instruction and classroom management. At PLC IV- Proficient/Accomplished Phase, candidates engage in increased field experience hours in P-12 schools, and they engage in student teaching where they plan, deliver, and assess instruction in the specialty area (2.4.a.1, 2.4.a.2, 2.4.a.3).

The candidate assessment system for advanced licensure programs also adheres to four transition points (2.4.a.1). Candidates are admitted to the program in Transition Point I-Admission to Program Candidacy. At Transition Point II-Midpoint Assessment, the unit collects data on candidate cumulative grade point average and comprehensive examination performance. In Transition Point III-Program Completion and Exit, the candidate constructs an electronic portfolio (e-portfolio) and engages in clinical practice. The Unit collects data on its alumni in Transition Point IV-Post Graduation to assess program quality and effectiveness (1.4.i, 1.4.j, 1.4.k).

Faculty constructed rubrics to assess electronic evidences and instrumentation to gauge candidate dispositions and pedagogical skills in P-12 schools (1.4.c.1, 1.4.c.21.4.c.4). To ensure fidelity in the data collection process, the Unit constructed assessment maps that illustrated the four transition points and key assessments for initial and advanced programs (2.4.a.1 p 23-25. Based on these assessment maps, the Unit compiles, aggregates, and disaggregates data from internal and external sources on applicants, candidates, and graduates to improve candidate performance and Unit operations. The Unit requires all initial and advanced licensure candidates to produce an e-portfolio in Taskstream that meets institutional, state, and professional standards. Initial licensure candidates produce and upload at least six evidences in the e- portfolio, and advanced licensure candidates produce and upload at least two evidences in the e-portfolio that faculty evaluate using the appropriate approved rubric (1.4.c.11.4.c.21.4.c.3).

Both initial and advanced candidate performance is formally reviewed at least once a year to monitor progress in the teacher education program. Results of these data-driven evaluations inform recommendations for candidate’s matriculation and thereby ensure greater opportunity for candidate success and program quality. This process sometimes results in students being recommended for transition out of the program.

The Unit practices shared governance in its teacher education program. The Chancellor appoints faculty representatives from various Schools and Colleges and P-12 public school partners are identified to serve on the Teacher Education Council (TEC), which consistently and systematically reviews data to influence policy (I.5.a.6). The Dean of the School of Education appoints faculty representatives from the Teacher Education Council to serve on the School of Education Assessment Committee. Committee members review key assessments and rubrics to ensure fairness, accuracy, and consistency (2.4.c.1).

The unit benchmarks all initial and advanced candidates annually. During the benchmarking process, the Dean’s Office assesses candidate performance based on transition point progress (2.4.c.1, 2.4.c.2). Candidates receive a letter from the Dean of the School of Education that articulates their current cumulative GPA, number of earned credit hours, and outstanding key assessments. This correspondence encourages candidates to visit their academic advisors to discuss other programmatic requirements or concerns. Summative results of the annual benchmarking are shared with the assessment committee, unit faculty, TEC members, and Dean’s Cabinet.

The Unit’s assessment system uses multiple electronic software platforms that allow for transparency and access. Faculty advisors and administrators can access individual candidate data from Aggie Access Online, or they may use Banner to download candidate performance outcomes (e.g., grade point average, course grades, earned credit hours) and personal data (e.g., demographics, standardized test scores) (2.4.d.4). Faculty and administrators can access programmatic data (e.g., retention data, program enrollment data) from the University’s Fact Book (2.4.d.6) which is maintained by the Office of Institutional Planning, Assessment, and Research. Unit administrators and faculty may formally request access to WebFOCUS (2.4.d.7) to generate comprehensive reports. Faculty, P-12 cooperating teachers, and students may access data from Taskstream (6.4.j.3) which serves as the primary repository of candidate performance data (e.g., dispositions data, teacher education interview data, electronic evidence scores, field experience and clinical practice data). A data analyst maintains a Microsoft Excel spreadsheet referred to as a “CSV” file, which houses key assessment data on each initial and advanced licensure candidate.

For the purposes of stakeholder transparency and accountability, CEd administrators use these electronic platforms to share the Unit’s annual progress. CEd administrators prepare annual assessment reports that illustrate initial and advanced candidate performance outcomes. CEd administrators post condensed summary data on the Unit’s website (2.4.d). During the TEC Data Institute (held at the beginning of the academic year), the Associate Dean of Undergraduate Programs shares elements of these reports and key summary data with faculty, public school partners, and other stakeholders (2.4.d). In addition to data collected in the unit, the SOE administrators and faculty consistently review data generated by state and national agencies such as the UNC Teacher Program Effectiveness studies, Educational Preparation data, and the National Council for Teacher Quality studies as well as NCATE-related data (1.4.k.2).

The Unit uses a number of processes to guarantee its assessment system is fair, consistent, and absent of bias. The CEd sponsors annual orientation sessions for initial and advanced licensure candidates where the administrators and faculty carefully explain the Unit’s assessment system and highlight key assessment tools that lead to licensure recommendation (2.4.d.5). Policies and processes are documented in various handbooks (2.4.a) including the Unit Assessment Handbook and the Field Experience and Clinical Practice Handbook (3.4.c, 3.4.e.1, 3.4.e.2). Syllabi in the professional education core courses include a synopsis of the Unit’s conceptual framework, learning outcomes, assessment and evaluation information, and the alignment of the course with state, national, institutional, and professional standards (I.5.b). The Assistant Dean of Student Support Services hosts an annual face-to-face training session with P-12 cooperating teachers, University supervisors, and teacher education candidates, and each person has access to support or tutorial documentation housed on the CEd website (3.4.d). Faculty from the Department of Educator Preparation who teach professional education core courses and evaluate key assessments in Taskstream engaged in a rubric training session during the 2012-13 academic year. This process was replicated with a smaller group of faculty who teach professional education core courses to establish inter-rater reliability for common electronic evidence rubrics. The inter-rater reliability coefficient (Cronbach’s Alpha) for Electronic Evidence 3 is .795 and .877 for Electronic Evidence 5.

The Unit also provides initial and advanced licensure candidates with opportunities to voice their concerns and complaints. The Unit’s Grade Appeal Policy articulates its grade appeal procedures, which requires candidates to communicate initially with faculty where they must provide documentation that supports their complaint. Those concerns then advance to the department chairperson. If the matter is unresolved, the concern and documentation move to the Unit’s Grade Appeal Committee comprised of faculty representatives across academic departments and student representation. Candidates who have other grievances have the right to meet first with the Unit’s Associate Deans where their concerns are documented and followed through until the matter is resolved. These approaches ensure candidates in initial and advanced programs are assessed and evaluated in a manner that ensures consistency, fairness, and absence of bias.

2.2.b Continuous Improvement

  • Summarize activities and changes based on data that have led to continuous improvement of candidate performance and program quality.
  • Discuss plans for sustaining and enhancing performance through continuous improvement as articulated in this standard.

The Unit uses data collected at the assessment system transition points to improve candidate performance and program quality. In 2011-12, the TEC Assessment Committee analyzed benchmarking data. At the initial level, the committee found that only 35% of education majors in PLC I passed the Praxis I examination. Data also showed that only 56 education majors registered for the Praxis during the 2010-11 academic year compared to 164 majors who registered for the test in the previous year. To address these issues, the SOE Dean created a Praxis I Committee that established a Praxis I Enhancement Plan (PEP) with detailed recommendations for immediate action to increase the number of students who take and pass the Praxis examination (1.4.d.30). The PEP was fully implemented during fall 2013. The unit also reinstituted the CUIN 102: Introduction to Teacher Education course to enhance new students' computational, written communication, and reading skills.

The TEC Assessment Committee analyzed data on the transition point from PLC I to PLC II and found that approximately 71% of education majors returned for their sophomore year. Faculty members met and designed a Living and Learning Community (LLC) for first-year new teacher education majors. This LLC, the DreamKeepers, was launched in fall 2012 (1.4.d.28).

During the 2011-12 academic year, the Teacher Education Council reviewed candidate disposition and teacher education interview data at the transition point from PLC II and PLC III. Data showed that candidate responses on the disposition instrument were consistently skewed in a favorable manner (i.e., "Strongly Agree" responses). Candidate representatives who serve on the Council revealed that students provide responses on this instrument that they believe faculty and administrators want to hear. Given these data, faculty will revise (fall 2014) the disposition instrument to address student response behavior. The Unit also transitioned the teacher education interview from a paper-based process to an electronic process where interviewers can input data in Taskstream immediately during a prospective candidate's interview.  This process improved the Unit's response rate to prospective candidates, alerting them of their performance. PLC 3 data showed that P-12 cooperating teachers needed additional clarification and support to assist them with understanding what candidates in early field experiences were allowed to do while in their classrooms. The Unit responded to these data by developing information video clips and PowerPoint presentations readily accessible to cooperating teachers on the SOE website. The Unit also created short video clips to assist cooperating teachers with navigating Taskstream to input candidate data on early field experience observation rubrics.

Prior to 2012-13, the Unit offered one general methods of teaching course (CUIN 640) for all initial licensure candidates. P-12 cooperating teachers, student interns, and University supervisors indicated that initial candidates required more focused instructional methods within their subject area. The TEC voted to return the methods courses to the content areas rather than offering one general methods of teaching course effective fall 2013 (2.4.d).

In a continued effort to improve candidate performance and program quality at the advanced level, the Dean's Cabinet, Teacher Education Council, and TEC Assessment Committee analyzed data along the transition points across programs. P-12 cooperating teachers provided feedback during a focus group activity in fall 2013 where they expressed concerns about early field experiences in program curricula for MAT candidates who do not have an undergraduate background in education. In fall 2014, the TEC Assessment Committee and TEC will examine and discuss this issue in greater detail to determine where additional field experience hours can occur in the program.

Data from the employer survey indicated that alumni from the Master of Arts in Teaching (MAT) Elementary Education program did not perform well with differentiating instruction in the classroom. ELED 619 Action Research and ELED 620 Teaching through Multiple Intelligences and ELED 620 were added to the program curriculum in fall 2013 to facilitate candidate understanding of how P-12 students learn differently.

The Master of School Administration (MSA) program revised its internship experience based on feedback from principals regarding program improvements. Prior to fall 2013, MSA candidates completed a 300-hour internship experience, but P-12 principals from partnering districts expressed that it was not as directed or guided as it should be. Beginning fall 2013, the Unit reorganized the 300-hour experience to require candidates to identify six different field-based experiences for each of the seven North Carolina School Executive Standards.

2.3 Discuss plans for sustaining and enhancing performance through continuous improvement as articulated in this standard.

To ensure continuous improvement, the Unit will sustain initiatives and procedures that enhance candidate performance and unit operations. These plans include the following:

  • A self-study of the Unit's data revealed a potential issue with the reliability, consistency, and fairness of data and a need to crystallize the data collection and analysis procedures.  The SOE Dean engaged the Vice Chancellor of Information Technology, the University Registrar, and the Vice Chancellor of Institutional Research in discourse to explore the possibility of establishing a more efficient technological data collection and data sharing process. In fall 2014, the Unit will shift from storing data only in Taskstream and spreadsheets to storing and sharing data between Taskstream and Banner. Faculty and administrators will then have the ability to extract spreadsheets from Web Focus that align with candidate admission, key evidences, licensure, test outcomes, and graduation (2.4.d.1).
  • The Office of the Registrar and Vice Chancellor of Information Technology will issue a pre- candidacy code in Banner that will transition to a candidacy code once the candidate is admitted to teacher education. This action will aid faculty when advising candidates.
  • The unit has reorganized its administrative and support personnel by establishing the following unit level positions to support the assessment system: (1) Associate Dean for Assessment and Graduate Programs, (2) Director for Digital Learning, Planning, and Assessment, and (3) Data Management Specialist. These positions have administrative and data responsibilities regarding the collection, analysis, development, warehousing, management, and utilization of candidate and unit data for program improvement. The plan is to maintain these positions as operational components of the unit.
  • The unit will continue the annual TEC Data Institute with faculty and P-12 school partners which occur at the beginning of the academic year.
  • Department chairpersons currently evaluate faculty using evaluation instruments that are discipline specific. The unit has begun developing a uniform faculty evaluation instrument to ensure a more standardized faculty assessment system across the unit to provide a more equitable faculty reward system. Implementation goes into effect during the 2014-2015 academic year.
  • The unit dean will secure resources from the Provost's Office that will become a permanent part of the budget for policies and practices that support unit and candidate assessment initiatives.
Back to Top