In-Depth Program Review Process

  • The process for in-depth program review was developed in the 2016 - 2017 school year and refined throughout the 2017 - 2018 school year. To help ensure a clear understanding of the process elements, a process diagram was developed and reviewed on a regular basis. Major elements of this image are further described below:

     Program Review Process
    Curriculum Writing to "Deep Dive"

    Given the time and effort invested in curriculum writing at Pine-Richland from 2014 - 2016, it is important to understand the relationship of that work to the in-depth program review process. The two-year curriculum writing process was designed to capture the current content in a consistent format through vertical teams (e.g., units, big ideas, and learning goals). That process allowed the department to identify strengths and opportunities for improvement. Most of the attention was directed internally at a review of our district’s current structure and practices.

    The in-depth program review process has a broader focus on all elements of the department. Importantly, the process was designed to emphasize a balance of internal needs and a review of best practices from external sources. It asks questions, such as, “Are we doing the right things?” or “Do we need to consider more significant changes in program design?” In the image above, the curriculum writing process is like a “springboard” to “dive” more deeply into the content area. The personnel, structure, and work were organized into four major subcommittees.

    Committee Composition and Structure

    We strongly believe that meaningful and lasting change requires the engagement of all key stakeholders. Since the in-depth process was being developed and implemented at the same time, the first organizational decision was the use of a core team and an expanded team. The core team included several district office administrators, building principals/assistant principals based on vertical team assignment, and a small group of academic leadership council members (i.e., department chairs) and teachers. The core team conducted the planning and thinking necessary to maximize the efficiency and effectiveness of the expanded team. The expanded team included all core team members and additional teachers to ensure representation by all buildings, levels, and courses. Although this was a larger group, it was still a small representation of the overall business & computer science department.

    Within the expanded team, members were then organized by four main subcommittees: (1) Research; (2) Exemplar K-12 Schools; (3) Connections to Universities, Businesses, and the Community; and (4) Data and Information. While each subcommittee was responsible for specific tasks, two overarching elements were critical. First, the arrows on the left side of the subcommittees indicate that the groups must collaborate and exchange information (i.e., no silos). Second, the arrows on the right side of the subcommittees demonstrate that key findings/learning were captured and organized by major research buckets.  

    It is important to note that the expanded teams also used a systematic approach to listen to students and parents. Student focus groups were organized at the high school, middle school, and Eden Hall. These groups were representative of the student body and a wide range of academic rigor. In addition, parent and community input was gathered during day and evening town hall sessions. Parents who were unable to attend those face-to-face meetings were able to submit comments electronically.

    Research “Buckets”

    Within each discipline, five key areas of investigation were identified to guide the work of the subcommittees. As business and computer science information was gathered by subcommittees, it was organized into five key “buckets”: (1) Financial Literacy; (2) Computational Thinking; (3) Career Exploration; (4) Grade-span Competencies and Skills; and (5) Emerging Trends. In the early months of the process, the “buckets” were dynamic, meaning that some initial concepts were removed or combined with other key themes. As the expanded team continued to learn, those titles were then finalized. Importantly, the arrows on the bottom of the buckets also demonstrate the relationship between areas (i.e., no silos). The subcommittees’ learning and identification of information for the buckets were interconnected, as information from one area informed others. Based upon the information gathered through the bucket findings, a set of emerging recommendations was developed.

    Emerging Recommendations

    A systems thinking approach was critical to the in-depth program review process. The transition from “findings” to “emerging recommendations” required skills of synthesis, critical thinking, healthy debate, and communication. The entire expanded team used one set of lenses to review the list of internal strengths and weaknesses.  The lenses refer to the four subcommittees. Some emerging recommendations were designed to improve current gaps and weaknesses. Other emerging recommendations were identified in the analysis of exemplary programs, universities, businesses, or in the research literature. The team brainstormed recommendations by identifying recurring themes, ideas, and opportunities for growth.  The team discussed, modified, and edited the recommendations.  Emerging recommendations were consolidated into a draft.  The expanded team worked with the draft to link the emerging recommendations to data provided by the subcommittees. 

    Balancing Priorities and Resources

    As a system, the “ripple effect” of recommendations was built into the process model. The team then put the emerging recommendations into the action-priority matrix.  The action-priority matrix evaluates the impact versus the effort of the emerging recommendations. Examining the use of people, time, and money allows for the identification of which recommendations were quick fixes, major projects, fill-ins, and hard slogs. For example, a hard slog was used to categorize those recommendations that would require much effort but have little impact on student learning.  The team then identified the final emerging recommendations.


    Figure 3

    Action Priority Matrix

    Figure 3: Elmansy, Rafiq. “Time Management Tips for Designers: The Action Priority Matrix.”Designorate, 14 June 2016, Accessed 14 Mar. 2017.

    Continuum of Improvement

    Throughout the in-depth program review process, it was important to maintain perspective on the nature of program improvements. Especially when considering effective elements of exemplary schools or programs, the desire to move from the current program ("Point A") to an ideal future ("Point Z") is natural. However, it is more realistic to recognize that meaningful program improvement within an organizational system will often result from a series of smaller steps ("Points B, C, D, etc."). Although depicted as a straight line in the image below (Figure 4), the in-depth program review committee recognizes that continuous improvement is not always a linear process.

    Figure 4 Arrow from Point A to Point Z