California Mathematics & Science Partnership (CaMSP ... · PDF fileCalifornia Mathematics &...

πCalifornia Mathematics & Science Partnership (CaMSP)

Statewide Evaluation

Year Eleven ReportCohort 10

Through September 30, 2015

Authors:

Patricia O’DriscollMikala L. Rahn, PhDAndrew Thomas, PhD

Lorie Sousa, PhDVelette BozemanJessica BognerAlbert Chen

Public Works90 N. Daisy Ave.

Pasadena, CA 91107(626) 564-9890

www.publicworksinc.org

California Mathematics & Science Partnership (CaMSP)Statewide Evaluation

Year Eleven ReportCohort 10

Through September 30, 2015

Authors:

Patricia O’DriscollMikala L. Rahn, PhDAndrew Thomas, PhD

Lorie Sousa, PhDVelette BozemanJessica BognerAlbert Chen

Public Works90 N. Daisy Ave.

Pasadena, CA 91107(626) 564-9890

www.publicworksinc.org

CaMSP 2015 Report Page i

Public Works

Table of Contents

Abstract ............................................................................................. iii

Section 1: Introduction & Evaluation Methods ......................................3

Section 2: State & Local Evaluation Design .........................................11

Section 3: Components Supporting STEM Learning .............................17

Section 4: Evaluation Results Overview .............................................31

Section 5: Statewide Measures & Outcomes ......................................53

Section 6: Conclusion & Next Steps ...................................................69

Appendices

Appendix A: Cohort 10 Map ...................................................... A-1Appendix B: Bibliography ..........................................................B-1

Page ii CaMSP 2015 Report

California Department of Education

CaMSP 2015 Report Page iii

Public Works

AbstractThe California Mathematics and Science Partnership Program (CaMSP) was established through Title II Part B of the Improving Teacher Quality Grant Programs of the No Child Left Behind (NCLB) legislation, the 2001 reauthorization of the Elementary and Secondary Education Act (ESEA). CaMSP is administered by the California Department of Education (CDE) Science Technology Engineering and Mathematics (STEM) office.

The CaMSP program is designed to impact student learning in mathematics and science by enhancing the content knowledge and teaching skills of classroom teachers through professional development activities. Partnerships developed between high-need Local Education Agencies (LEAs), defined by 40% or higher participation in the National School Lunch Program (NSLP), and the mathematics, science and engineering faculty of Institutions of Higher Education (IHEs) are core to the improvement efforts sought by the CaMSP program. County Offices of Education (COEs) and other organizations concerned with mathematics and science education may also participate in partnerships. Individual partnerships that are part of the CaMSP program are designed to serve as models that can be replicated in educational practice to improve the mathematics and science achievement of California students.

Within the guidelines of CDE’s request for applications, each funded partnership was developed to meet local needs and take advantage of opportunities for collaboration with IHEs while at the same time meeting statewide goals and operational principles. In their applications for funding, partnerships determine the number of districts, schools and teachers targeted and must serve a cohort of a minimum of 30 teachers for the duration of funding in order to maintain the CaMSP grant. Through a request for application process that has evolved over time, CDE has funded partnerships under separate cohorts based on available funding.

This evaluation report focuses on implementation during 2014-15 and includes qualitative data and analysis of teacher content assessment and student outcome data collected from Cohort 10 partnerships. For the STEM Office, the 20 partnerships funded under Cohort 10 represented a substantial shift in focus of CaMSP to a STEM approach, where partnerships were directed to design professional development models for grades K-12 that integrated the disciplines of science, technology, engineering, and mathematics (STEM) through a variety of models for and approaches to professional development.

This report has been prepared by Public Works (PW), a nonprofit corporation based in Pasadena, California, which was selected by CDE through request for proposal as the statewide evaluator. The statewide evaluation incorporates both process measures focused on how the program is implemented and outcome measures focused on the results of the program and interventions as designed by individual partnerships funded by CaMSP. Using both qualitative and quantitative data collection methods, the evaluation includes site visits, telephone interviews, observations of professional development, statewide partner and teacher surveys, analysis of teacher demographic and participation data, and a student outcome study designed to evaluate the effect of CaMSP on student achievement in mathematics and science.

Under the statewide evaluation, CDE has incorporated the additional role of technical assistance to partnerships and support of the local evaluation, a federally required component of the grant program. For each partnership funded under the federal MSP program, a local evaluation must be conducted, which includes annual and summative evaluation reports and the completion of an Annual Performance Report (APR) online. Each partnership must also report on two Government Performance Reporting Act or GPRA measures: (1) state assessment results for students of participating teachers, and (2) participating teacher pre- and post- teacher content knowledge assessment results.

Page iv CaMSP 2015 Report


Under Cohort 10 and subsequent Cohort RFAs, CDE centralized implementation of the local evaluation under PW to include all required elements and a customized local evaluation design to support implementation and communication with partnership stakeholders at the local level. Under PW, local evaluations were also designed to support partnership reporting at the state level.

CaMSP Key Features

Partnership-Driven: The partnership represents its target population of districts, teachers and institutions of higher education. All partners exhibit a high level of commitment to the partnership. The governance structures include both the trainers/IHE and the districts as equal partners in planning the curriculum and logistics. The target population of the grant is served.

Teacher Quality: The partnership has created a cohort of teachers enrolled in all aspects of the professional development. They are on their way to meeting the target number of teachers involved and the hours required. The approach to professional development is tied to state standards and is focused on both improving the content knowledge and pedagogical approach of teachers. The professional development is research-based, high quality for both intensive training and followup components that include monitoring of implementation.

Challenging Courses & Curricula: Professional development is aligned to standards and aimed at transforming and improving instruction. The project is creating new challenging courses, lessons and curricula for pre-service or existing teachers and/or students. New courses, curricula, expectations or experiences for students will result from the professional development that teachers receive in this grant.

Institutional Change & Sustainability: The role of the IHE is clear and integral to the project. The IHE is involved in the planning, curriculum development and delivery. The education department and the discipline department (mathematics/science) are involved. Teachers can receive credit for their professional development. The institutions (IHE and Districts) are impacted by the project in terms of a tangible result. There are sustainable elements of the grant.

Evidence-based Design & Outcomes: Partnership uses a research or evidence-based model for professional development. The evaluation plan makes sense for the project. The design and measurement system will produce an impact on teacher and classroom quality. There is a cohort of teachers for which examining the impact on student outcomes makes sense. A pre-/post- assessment of teacher knowledge is conducted and local student assessment is conducted (above and beyond CSTs).

Organization of the Report This evaluation report is divided into six sections and an appendix.

Section 1: Introduction & Evaluation Methods: The first section of the report is an introduction to the CaMSP program and provides an overview of partnerships funded under CaMSP, program requirements, and the key features of CaMSP under the transition to Science, Technology, Engineering and Mathematics (STEM).

Section 2: State & Local Evaluation Design: This section of the report describes the statewide evaluation methods and the transition to coordination of the local evaluation under Public Works starting with Cohort 10.

CaMSP 2015 Report Page v

Public Works

Section 3: Components Supporting STEM Learning: This section of the report provides a summary of the literature review conducted for the statewide evaluation describing the background and rationale for integrated STEM learning and the kinds of structures and implementation strategies that can support its growth.

Section 4: Evaluation Results Overview: This section of the report provides a summary of implementation including site visits, professional development observations, teacher focus groups, partnership director telephone interviews and teacher and partner survey results.

Section 5: Statewide Measures & Outcomes: This section includes an analysis of data collected for the two required state measures, which include a teacher content assessment and student test scores in state assessments for mathematics and science. This first student outcome study for Cohort 10 includes a comparison of treatment teachers to a matched comparison group of non-participating teachers and regression analysis results.

Section 6: Conclusion & Next Steps: This section concludes the report with a summary of the data collected and a description of next steps in the evaluation.

Page vi Introduction & Evaluation Methods


x2 • y2 = z2

F=ma

Al2(SO4

)3

79

Au197.0

F = Gm1 m

2 /r 2

A = πr2

Section 1:

Introduction & Evaluation Methods

CaMSP 2015 Report Page 3

Public Works

Section 1: IntroductionThe California Mathematics and Science Partnership Program (CaMSP) was established through Title II Part B of the Improving Teacher Quality Grant Programs of the No Child Left Behind (NCLB) legislation, the 2001 reauthorization of the Elementary and Secondary Education Act (ESEA). CaMSP is administered by the California Department of Education (CDE) Science Technology Engineering and Mathematics (STEM) office.

Through a competitive Request for Proposal (RFP) process, CDE hired Public Works (PW), a non-profit consulting company in Pasadena, California, to be the initiative’s third-party evaluator. The results in this annual report reflect activities from Year 11 and data collected during the 2014-15 school year, highlighting the transition to Science Technology Engineering and Mathematics (STEM) professional learning supported by CaMSP. Previous reports and additional information about CaMSP are available on the Public Works Website (www.publicworksinc.org).

Under the MSP program of the US Department of Education and through CaMSP, teachers have participated in professional development activities since 2003 that focused on evidence-based science and mathematics teaching methods to improve teacher content knowledge and student academic achievement. Participants in CaMSP must demonstrate a sustained commitment to professional development over three years and are required to complete 60 hours of intensive professional development (e.g. summer institutes, one day trainings during the school year) and 24 hours of classroom followup activities such as lesson study, professional learning communities and coaching in which what is learned during the intensive professional development is embedded in the classroom.

From 2003 to 2013, the California Department of Education (CDE) funded 135 CaMSP partnerships that encompassed multiple districts, institutions of higher education (IHE), county offices of education (COE) and other professional development partners across the state. These 135 partnerships were funded by

State and Federal Administration of CaMSP

The Professional Learning Support Division Science, Technology, Engineering and Mathematics (STEM) Office of the California Department of Education (CDE) is responsible for administering the competitive grant program. Less than three percent of the state’s allotment is expended for administration of the program. These funds support the external evaluation, application competition and the Learning Network.

At the end of an initial funding cycle, partnerships may submit a review of their progress toward program objectives and, if successful, receive additional funding. Partnerships are required to submit a Project Profile, Project Narrative and Local Evaluation Report directly to the federal government through its Annual Performance Reporting System (APR).

Under CaMSP, professional learning opportunities must be designed to: • Improveteachers’subjectmatterknowledge.• Relatedirectlytothecurriculumandacademicareas

in which the teacher provides instruction. • Enhancetheabilityoftheteachertounderstand

and use the challenging California academic content standards for mathematics and science.

• Provideinstructionandpracticeintheeffectiveuseof content-specific pedagogical strategies.

• Provideinstructionintheuseofdataandassessments to inform classroom practice.

and may incorporate: • Opportunitiesforteacherstoworkcollaboratively

with experienced teachers, college faculty or business professionals.

• Leadershipdevelopmentactivitiestoidentify,develop, and employ exemplary mathematics and science teachers as professional development providers.

• Professionallearningactivitiesthatincludeadditional activities, such as curriculum alignment, distance learning, and activities that instruct teachers in the appropriate use of technology in the classroom.



three-year professional development grants over 11 separate cohorts. These included Cohorts 1 through 9, a Research cohort and a Demonstration cohort, and were focused on mathematics or science. These cohorts were limited to grades 3 to 8 for science and 3 to Algebra I for mathematics.

For the STEM Office, the twenty partnerships funded under Cohort 10 represented a substantial shift in focus of CaMSP to a STEM approach where partnerships were directed to design professional development models for grades K-12 that integrated the disciplines of science, technology, engineering, and mathematics through a variety of models for and approaches to professional development. In 2015, the STEM Office issued two subsequent RFAs to fund additional partnerships. Cohort 11 began in early 2015 and included 12 partnerships modeled largely after the structure of Cohort 10. In May 2015, 12 additional partnerships were funded under Cohort 12, which retained more of an emphasis on discipline-specific professional development to support implementation of new mathematics and science standards adopted by California under the Common Core State Standards (CCSS) initiative for mathematics (or CCSS-M) and English language arts and the Next Generation Science Standards (NGSS). Data collected for these two new cohorts will be included in the next statewide evaluation report, which incorporates implementation of CaMSP in the 2015-16 school year.

CaMSP Program RequirementsThe purpose of the CaMSP program is to improve the mathematics and science achievement of students by encouraging LEAs and IHEs to form eligible partnerships that:

• SupportmathematicsandsciencecurriculaalignedwithCaliforniaacademiccontentstandardsand implement kindergarten through 8th grade instructional materials adopted by the State Board of Education (SBE).

• ImprovemathematicsandscienceteachingbyencouragingIHEstoassumegreaterresponsibility for improving mathematics and science teacher education through a comprehensive system of teacher preparation that guides and advises mathematics and science teachers.

• Focusontheeducationofmathematicsandscienceteachersasacareer-longprocessthatcontinuously stimulates teachers’ intellectual growth and upgrades teachers’ knowledge and skills.

• BringmathematicsandscienceteacherstogetherwithIHEfaculty,aswellasscientists,mathematicians, and engineers to mutually increase subject matter knowledge and improve instructional strategies.

• Makeevidence-basedcontributionstotheteachingandlearningknowledgebasetoinformtheunderstanding of how students effectively learn mathematics and science.

CDE has awarded the MSP funds in separate cohorts of partnerships. In each cohort, partnerships apply and compete for funds based on a Request for Applications (RFA). The RFAs released by CDE have been changed and refined for each cohort of partnerships. Partnerships applying for a CaMSP grant must include:

• Ahigh-needLEAand• Anengineering,mathematicsorsciencedepartmentofanInstitutionofHigherEducation

(IHE).

Defined by each state, the term “high-need LEA” in California refers to an LEA where at least 40 percent of the students it serves qualify for the National School Lunch Program (NSLP). Partnerships may also include:


Public Works

• AdditionalLEAs,includingCOEs,publiccharterschools,publicorprivateelementaryschoolsor secondary schools or a consortium of such schools;

• Anotherengineering,mathematics,scienceorteachereducationdepartmentofanIHE;• Abusinessorindustryorganization;• Anonprofitorfor-profitorganizationofdemonstratedeffectivenessinimprovingthequalityof

mathematics and science teachers; • Publicorprivateorganizations,agencies,andfoundations;and• Localparentorganizations.

The CDE review and grant funding process includes an initial funding cycle of 18 months and, if approved for continuation, funding is available for an additional two 12-month cycles. During these cycles, the same group of participating teachers is required to complete 84 hours of CaMSP professional development (60 hours of intensive and 24 hours of classroom followup) in each funding cycle in order to continue as part of the partnership.

In accordance with NCLB Section 2201, the CaMSP program is governed by the Uniform Provisions Act and requires the equitable participation of teachers who teach in nonprofit private schools located in districts where grants are awarded. Prior to submitting a grant request, each LEA in the partnership must engage in timely and meaningful consultation with representatives of private schools regarding the needs of their teachers related to improving mathematics and science teaching.

All partner organizations share responsibility and accountability for the CaMSP program. Each partner organization is required to provide evidence of its commitment to undergo the coordinated institutional change necessary to sustain the partnership effort beyond the funding period. Community colleges are encouraged to participate in CaMSP because of the strong role they play in the preparation and professional development of a diverse mathematics and science teacher workforce.

The Lead Partner in a CaMSP partnership must be a high-need LEA. The Lead Partner submits the partnership proposal and signs the grant award assurances to accept management and fiduciary responsibility for the partnership. A Leadership Team must be convened and meet regularly to oversee the development of the program and the administration of the CaMSP.

Cohort 10 and the Transition to STEMCohort 10 partnerships had to choose at least one core discipline (mathematics and/or science) and at least one supporting discipline (engineering and/or technology) to provide content instruction, pedagogical strategies and curriculum development support to participating teachers. In addition, Cohort 10 partnerships were directed to support implementation of California’s newly adopted standards for mathematics and science.

Each partnership in Cohort 10 had to incorporate strategies for participating teachers to determine local gaps in STEM learning, embed technology, experience how to prepare students for STEM career and college options and become teacher leaders within their schools and districts. Goals for students under Cohort 10 included improving performance on state mathematics and science assessments, learning about STEM college and career choices and how STEM is incorporated in everyday life. The following table provides information about Cohort 10 from the STEM Office CaMSP Request for Applications (RFA):



Cohort 10 STEM Definition:An approach to teaching and learning that emphasizes integral connectedness of at least one of the core disciplines—science and mathematics—and at least one of the supporting disciplines—technology and engineering. The connections are made explicit through the collaboration of both educators and their students, resulting in real and appropriate contexts that are built into the instruction, curriculum and assessment. The common element of problem solving as defined in the CCSS-M and the NGSS is emphasized across the identified STEM disciplines and grade spans allowing students to engage, explore, expand and evaluate their learning and apply critical thinking skills as they learn.

STEM-ED Objectives for teachers: • CreateSTEMStructure• Determinegaps• EngageinappropriatePD• Learnnewteachingstrategies• LearnandexperienceSTEMcollegeand

careers• Integrate• Create• Pilot• ImplementnewstandardsinaSTEMcontext• Becometeacherleaders(byYear3)• Facilitateimprovedstudentperformancein

STEM-ED disciplines

STEM-ED Objectives for students: • Learnacrossdisciplines• Integrateacrossdisciplines• ExperienceSTEM• Engage• Inquire• Improveandsucceed• DemonstratepositiveattitudestowardSTEM

CaMSP Key Features and the Transition to STEMUnder the NCLB legislation and the MSP program, CaMSP partnerships have focused on understanding baseline student achievement and teacher workforce data in order to document improvements in student achievement in mathematics and science over time. In addition, CaMSP partnerships must be designed to address the following five key features: (1) partnership-driven, (2) teacher quality, (3) challenging courses and curricula, (4) evidence-based design and (5) outcomes and institutional change and sustainability. For Cohort 10, the STEM Office RFA further refined how partnerships were to approach the five key features. The shifts in each of these features under Cohort 10 are described below each key feature description.

(1) Partnership-Driven: Programs are designed and implemented by partnerships that unite administrators, teachers, and guidance counselors in participating LEA partner organizations and disciplinary faculty, education faculty, and administrators in IHE partner organizations. Partnerships draw upon the expertise of all members to meet the purposes of this program. Scientists, mathematicians, engineers, and individuals from other partner organizations, including COEs, may also play significant roles in program activities. Partners are deeply engaged in the effort at both the institutional and the individual levels and share goals, responsibilities, and accountability for the program.

Cohort 10: Change from a single leadership team to a Core partnership made up of the lead LEA, participating LEAs, California-based IHE, and PD providers and a Regional Collaborative Partnership (RCP) to serve in an advisory and supportive role with the goal to extend the benefits of the project regionally (includes K-16 partners, business/industry, government, chambers of commerce, COE’s, community-based organizations).

(2) Teacher Quality: Programs enhance the quality and expertise of teachers who teach mathematics and science. Drawing upon the expertise of mathematics, science, and engineering faculty in IHE partner organizations, teachers are engaged in high quality professional development activities to develop strong


Public Works

mathematics and science content knowledge and related pedagogical strategies. Program activities ensure that educators develop the necessary knowledge and skills to teach challenging courses effectively using State Board of Education (SBE)-adopted standards and instructional materials. Partnerships also develop and implement innovative strategies that include increasing the diversity of the teacher workforce and encouraging young women and other underrepresented individuals to achieve in mathematics and science.

Cohort 10: Expanded qualifying grade levels to include Kindergarten through 12 under the following grade spans: K-2, 3-5, 6-8, 9-12 or two contiguous grade spans. In addition, the core partnership is responsible for identifying new STEM education content/programming to pilot in participating teacher classrooms with expansion in Year 3 to other schools and districts. Participating LEAs were also asked to provide assurance that participating teachers would remain in the assigned grade span throughout the duration of the grant.

(3) Challenging Courses & Curricula: Programs ensure that students are prepared for, have access to, and are encouraged to participate and succeed in challenging mathematics and science courses. Various approaches that integrate reasoning, problem solving, hands-on and procedural skills are applied. Evidence of high quality professional development is demonstrated by the instruction of well-trained teachers whose students have access to high-level mathematics and science content. Classroom instruction at all grade levels incorporates appropriate levels of rigor and challenge building on skills from one level in preparation for the next.

Cohort 10: This feature has been expanded and identified as the development of specific “products” under Cohort 10, which includes the development of integrated curriculum, units and other programmatic strategies to blend STEM education in courses or programs of study. Professional development is designed to develop, pilot and disseminate these products in Year 3. Cohort 10 also specifies integration of mathematics and science standards through problem solving and critical thinking.

(4) Evidence-Based Design & Outcomes: Program design is informed by current research. Program outcomes should contribute to the knowledge base of teaching and learning. Through participation in the California and National MSP Learning Networks, programs will collectively contribute to the knowledge base on teaching and learning so that research findings and successful evidence-based strategies can be broadly disseminated to improve educational practice. Programs also link assessment (classroom, local, and state) and accountability measures to their design and outcomes.

Cohort 10: Under this cohort, the lead LEA and core partnership LEAs have been asked to work directly with Public Works to provide teacher, student and school data for the local and state evaluations. In addition, each partnership contracts directly with Public Works to assist in the design of a local evaluation plan that incorporates customized measures aligned to the partnership’s approach and inclusion of the Regional Collaborative in the evaluation.

(5) Institutional Change & Sustainability: To ensure program sustainability, partner organizations leverage resources and design and implement new policies and practices leading to well-documented, inclusive, and coordinated institutional change at both the IHE and the LEA level. IHE partner organizations commit to engaging mathematics, science, or engineering faculty, or any combination thereof, in activities that strengthen their teaching practices and their roles in mathematics and science education, including teacher preparation and professional development. Partner organizations commit to providing environments for teachers, guidance counselors, and administrators that support an evidence-based approach and in which exemplary contributions to mathematics and science learning and teaching are recognized and rewarded. Other partners commit to engaging mathematicians, scientists, engineers, and other individuals in activities that strengthen their roles in mathematics and science education for the long term.



Cohort 10: Project director assigned to the partnership must be an employee of the lead LEA for core partnership. In addition, a 50% match for the project director’s salary is required by the lead LEA, COE or consortium. Under this cohort, the project director may have an expanded role in the professional development by serving as a coach, facilitator or professional development provider. Cohort 10 emphasizes the importance of dissemination of the work of the partnerships and indicated that IHE’s must be California-based. For the core partnership, all disciplines selected for the partnership must be represented from the IHE partner or partners.

Organization of the Report This evaluation report is divided into six sections and an appendix.

Section 1: Introduction & Evaluation Methods: The first section of the report is an introduction to the CaMSP program and provides an overview of partnerships funded under CaMSP, program requirements, and the key features of CaMSP under the transition to Science, Technology, Engineering and Mathematics (STEM).

Section 2: State & Local Evaluation Design: This section of the report describes the statewide evaluation methods and the transition to coordination of the local evaluation under Public Works starting with Cohort 10.

Section 3: Components Supporting STEM Learning: This section of the report provides a summary of the literature review conducted for the statewide evaluation, describing the background and rationale for integrated STEM learning and the kinds of structures and implementation strategies that can support its growth.

Section 4: Evaluation Results Overview: This section of the report provides a summary of implementation, including site visits, professional development observations, teacher focus groups, partnership director telephone interviews and teacher and partner survey results.

Section 5: Statewide Measures & Outcomes: This section includes an analysis of data collected for the two required state measures, which include a teacher content assessment and student test scores in state assessments for mathematics and science. This first student outcome study for Cohort 10 includes a comparison of treatment teachers to a matched comparison group of non-participating teachers and regression analysis results.

Section 6: Conclusion & Next Steps: This section concludes the report with a summary of the data collected and a description of next steps in the evaluation.

x2 • y2 = z2

F=ma

Al2(SO4

)3

79

Au197.0

F = Gm1 m

2 /r 2

A = πr2

Section 2:

State & Local Evaluation

Design


Public Works

Section 2: State and Local Evaluation Design Public Works (PW) has been selected as the statewide evaluator under three separate RFP’s issued by CDE to support administration of the program. As the statewide evaluator, PW provides an annual report to the STEM Office that includes both qualitative and quantitative analysis of program outcomes. In addition, PW maintains an attendance database for participants to document hours and types of professional development and collects state student outcome data to provide consistent reporting of one of two required Government and Performance Reporting Act (GPRA) measures for the MSP program.

In the most recent contract with PW, CDE included an additional role of technical assistance to partnerships and support of the local evaluation, a federally required component of the grant program. For each partnership funded under the federal MSP program, a local evaluation must be conducted, which includes annual and summative evaluation reports and the completion of an Annual Performance Report (APR) online.

In this transition to local evaluations coordinated by PW, the STEM Office also requested the use of consistent measures of teacher content for both mathematics and science, which is the second required GPRA measure for the MSP program. To align as closely as possible with the spirit and intention of CaMSP and the transition to new mathematics and science standards in California, PW selected the Learning Mathematics for Teaching or LMT measure for mathematics-focused partnerships. Because of the lack of consistent and/or appropriate measures of teacher content knowledge in science, PW developed an NGSS-aligned Teacher Content Assessment in Science (TCAS) to be implemented under Cohort 10.

Under Cohort 10 and subsequent Cohort RFAs, CDE centralized implementation of the local evaluation under PW to include all required elements and a customized local evaluation design to support implementation and communication with partnership stakeholders at the local level. Under PW, local evaluations were also designed to support partnership reporting at the state level.

Statewide Evaluation Research Questions & Data CollectionUsing both qualitative and quantitative data collection methods, the statewide CaMSP evaluation focuses on the following research questions:

• Howhavethepartnershipsensuredthatallstudentshaveaccessto,arepreparedforandareencouraged to participate and succeed in challenging and advanced mathematics and science classes?

• Howhavethepartnershipsenhancedthequalityofthemathematicsandscienceteacherworkforce?

• Whatevidence-basedoutcomesfromthepartnershipscontributetoourunderstandingofhowstudents effectively learn mathematics and science?

Each year, the PW evaluation collects information using the following data collection strategies: (1) partnership telephone interviews and site visits, (2) a statewide survey of participating partners and teachers, and (3) a teacher and student outcome study using a quasi-experimental model that compares students of CaMSP treatment teachers to a matched comparison group of non-participating teachers.

For this report, PW conducted an initial site visit and planning meeting with Cohort 10 partnerships in 2014, a telephone interview in the Spring of 2015 and visited each partnership during the Summer of 2015 to observe professional development and conduct a focus group of participating teachers. PW

State & Local Evaluation Design


also conducted a survey of Cohort 10 Core partners and Regional partners in Fall 2014 and a survey of participating teachers in Spring 2015.

The statewide evaluation also includes results of the first student outcome study for Cohort 10, which incorporates Spring 2015 results from the California Standards Test for science (grades 5, 8 and 10) and the first administration of the Smarter Balanced Assessment Consortium (SBAC) in mathematics (grades 3 to 8 and 11).

Annual Site Visit to a Subset of Partnerships: Each year, PW visits a subset of partnerships. Using structured site visit protocols and methodology PW conducts interviews and focus groups of key stakeholders and observes professional development or leadership team activities offered by the partnerships. PW collects data from multiple partners related to the five key features of the program: partnership-driven, teacher quality, challenging courses and curricula, evidence-based design and outcomes, and institutional change and sustainability.

Partnership Phone Interviews: Each year, the evaluation includes a telephone interview of each project director in all currently funded partnerships that are not visited. This interview incorporates a structured interview protocol collecting information about the status of achieving partnership training targets and teacher retention, the professional development model, and how it has changed, evidence of student and/or teacher outcomes, sustainability, institutionalization, lessons learned, and future plans.

Partner and Teacher Surveys: PW administers a partner survey and a participating teacher survey each year. The partner survey is administered annually in the fall and focuses on the involvement and roles and responsibilities of the stakeholders or partners in each grant/partnership (including institutions of higher education and professional development providers). The participating teacher survey is administered each spring to every teacher participant in the grant who has participated in at least one hour of training. The survey asks teachers for their opinions about how CaMSP professional development has impacted their teaching practice.

Teacher Database: In order to measure the outcomes of CaMSP on participating teachers and their students, PW has developed a teacher database that includes teacher demographic data and is used by partnerships to maintain teacher participation/attendance data. PW uploads individualized teacher data from the Professional Assignment Information Form (PAIF) for all teachers at targeted grade levels in all schools included in the partnership (both participating and nonparticipating teachers). This confidential data provides PW with the necessary information to calculate a baseline level on indicators of teacher quality prior to participating in the grant and then each year thereafter. This data also allows for comparison of this information to nonparticipating teachers. PW has developed an online interface to this database and partnerships use this tool to provide information about teacher professional development attendance. Activities are labeled as either intensive or classroom followup and by clear descriptions such as “lesson study at Emerson School” or “one-day workshop for 7th grade science teachers,” as well as start and end dates, funding cycle, and funding level. Teacher confidentiality is maintained through strict internal protocols.

Student Outcome Data and Study: From the teacher database, PW has created lists of both treatment (teachers completing 84 hours of training) and control teachers (a matched sample based on PAIF data of teachers not participating in the training) for each cohort. From these lists, CDE and districts provided both baseline and achievement data for students on the California Standards Test (CST) in science and the Smarter Balanced Assessment Consortium (SBAC) in mathematics. For this study, teachers are matched using similar teacher characteristics and students are matched based on similar demographic characteristics and prior achievement.


Public Works

Teacher Content Assessment in Mathematics and/or Science: As required for the federal reporting on the Annual Performance Report, all teachers participating in CaMSP are administered a teacher content assessment in either mathematics or science administered in a proctored environment. Depending on the disciplines selected or the individual teaching assignment (primarily for secondary teachers), teachers complete pre- and post-assessments of either the Learning Mathematics for Teaching (LMT) or the Teacher Content Assessment in Science (TCAS) in each funding cycle. The LMT Project was founded in 2000 by researchers at the University of Michigan in an effort to build a questionnaire that measured pedagogical content knowledge within the domain of mathematics. The intent of the conception of the LMT was to create an instrument that could measure the effects of Professional Development on mathematics instruction. The Teacher Content Assessment for Science (TCAS) was developed by Public Works to provide a Next Generation Science Standards (NGSS)-based measure that could fulfill the need for a common assessment among CaMSP participants focused on science learning, and includes multiple choice and constructed response items from Earth, Life and Physical Science. Constructed response items include engineering problems designed to support measurement of the NGSS engineering practices. The pre-assessments were administered in the summer of 2014 and the post assessments were administered in summer of 2015 after a full year of exposure to the Cohort 10 partnership programs.

Local Evaluation Plans and ReportingAs both the state and local evaluator for Cohort 10 partnerships, Public Works conducted an initial site visit and planning meeting for the 20 partnerships during the spring and summer of 2014. In addition, PW was accompanied by local evaluation consultants with previous experience in CaMSP to collect and analyze site visit data and to provide initial support to projects in implementing local evaluation strategies. The visits were conducted with the core leadership team focused on the five key features of CaMSP described in the introduction: (1) partnership driven, (2) teacher quality, (3) challenging courses and curricula, (4) evidence-based design and outcomes and (5) institutional change and sustainability. In addition, site visits collected information regarding the status of implementation of CCSS-M and NGSS into grant activities.

The purpose of the first site visit and planning meeting was to check in at the initial stages of development to better understand the direction of the professional development and to develop a local evaluation plan as a blueprint to guide the partnership over the next three years. In the first phase of the evaluation plan, the focus is on instrumentation needed and on initial data collection in the summer and fall of the first year of implementation.

With the coordination of both the state and local evaluations by PW under Cohort 10, there is a strategy in place to collect similar data across all sites as well as to create customized instrumentation and data collection for

each partnership’s individual model and approach. Data from the state evaluation that is relevant at the partnership level is disaggregated and analyzed for each partnership. This includes the core and regional collaborative partner survey, the annual participating teacher survey, and state student outcome data including the SBAC for mathematics and CST for science.

In September 2014, evaluation plans developed initially from the partnership proposal and refined during the site visit were finalized with partnership input and submitted to CDE’s STEM Office as an attachment to the partnership’s first quarterly Year to Date (YTD) report. PW coordinated the development of the tools identified with each partnership and the collection of data for the customized

Local evaluation plans included a variety of methods including:

• Teacherself-efficacyandbeliefsurveys• StudentSTEMinterestsurveys• Studentassessmentembeddedin

curriculum products • Coachingandclassroomobservation

tools • Lessonstudyrubrics• Analysisofdatafromdistrictstudent

assessment systems.

State & Local Evaluation Design


components of the local evaluation and provided regular updates to CDE as part of the YTD reports that each partnership submits. In October 2015, PW provided the first evaluation report that was attached to each partnership’s APR and provided support to the partnerships in the completion of sections of the APR related to evaluation.

x2 • y2 = z2

F=ma

Al2(SO4

)3

79

Au197.0

F = Gm1 m

2 /r 2

A = πr2

Section 3:

Components Supporting STEM

Learning


Public Works

Section 3: Components Supporting STEM LearningThe transition to a STEM-focused approach to professional learning for teachers under CaMSP first implemented under Cohort 10 reflects at least two important understandings in improving schools. First, policy makers, advocates for educational equity and closing achievement gaps, and the business community continued to connect to and push publicly for improvements in the systems that prepare young people to become mathematicians, scientists, computer scientists and engineers. Second, the transition to STEM-focused professional learning for teachers combined what has been learned about high quality professional development with a sense that all students would benefit from a more integrated approach to learning in order to have the skills and knowledge base they would need to contribute to the robust and ongoing changes in technology and a complex economy. Numerous initiatives at the US Department of Education, other federal agencies and California’s own annual STEM Symposium reflect this attention to the need for more students to enter STEM fields, teachers to be trained in STEM learning and for schools to implement STEM as both specific courses or programs or a more universally integrated approach to curriculum and instruction that frames the entire school day.1

For California, this renewed attention to STEM occurred amidst vast changes to the accountability-based system of standards and assessments that had developed under the No Child Left Behind Act (NCLB), the 2002 reauthorization of the Elementary and Secondary Education Act. These changes included new standards and assessments for mathematics and English language arts and new science standards. With the recent signing into law of ESEA under the Every Student Succeeds Act (ESSA) on December 10, 2015, California is devising a transition plan and anticipating further changes to accountability systems at the federal and state levels. During this time, changes included:

• In2013,CaliforniaEducation Code Section 52056(a) requiring an Annual Performance Index (API) ranking of schools was repealed and the state enacted the Local Control Funding Formula (LCFF).

• In2014,theU.S.DepartmentofEducationapprovedCalifornia’stestingwaiverunderNCLBtestingprovisions, granting a one-year waiver that allowed flexibility in making federal Adequate Yearly

1 See the US Department of Education’s Science, Technology, Engineering and Math: Education for Global Leadership web page for more information (http://www.ed.gov/stem). See also http://www.stemcalifornia.org/ for information about the California Department of Education’s STEM Symposium.

• Common Core State Standards: In 2010, the State of California adopted the Common Core State Standards (CCSS) for both mathematics and English language arts. The Common Core State Standards were designed to show the knowledge and skills students will need to be prepared to enter their first year of college courses or the workplace with competitive salaries. The CCSS-M focuses on traditional mathematical areas: numbers and operations, algebra, functions, statistics and probability, and geometry.

• Smarter Balanced Assessment: With California adopting the CCSS, California joined the Smarter Balanced Assessment Consortium (SBAC) to develop statewide assessments that are aligned to the CCSS. The Smarter Balanced Assessment is administered to grades 3 through 8 and 11. The idea behind the SBAC is for students to move beyond multiple-choice questions to open ended problem solving performance tasks where they will be able to defend or explain their reasoning in writing. California participated in the field testing of the SBAC in 2014, with full implementation and scores reported from the SBAC in spring 2015 in the CAASPP system.

• Next Generation Science Standards: The Next Generation Science Standards (NGSS) adopted by California, seeks to develop deeper understanding of science for K-12 students. The standards ask students to apply what they learn through the practices of scientific inquiry and engineering design. The NGSS weaves together three dimensions: (1) disciplinary core ideas, (2) science and engineering practices, and (3) cross-cutting concepts to outline clear performance expectations.

Components Supporting STEM Learning


Progress (AYP) determinations for elementary and middle schools and elementary and unified school districts participating in the Smarter Balanced Field Test.

• In2014,thestatebegantoimplementanewaccountabilitysystemundertheCaliforniaAssessmentof Student Progress and Performance (CAASPP) System of assessments, which began with the English language arts and mathematics SBAC system and included the suspension of the California High School Exit Exam (CAHSEE) in 2015 under Senate Bill 172.2

A Historical Perspective on the Transition to STEM IntegrationWith renewed national attention on integrated STEM education and 21st Century Learning Skills, along with the emphasis on a more applied approach to mathematics and science education, CDE redesigned the CaMSP application to help local school districts collaborate with IHE faculty to meet these needs. Beginning with Cohort 10, the focus of CaMSP shifted to professional development in content and pedagogy aimed at the integration of mathematics and/or science with content, applications, processes, and experiences in engineering and/or technology. Newly adopted standards in mathematics and in science that emphasize pedagogical practices of inquiry, integration, and applied learning have prompted policymakers and educators throughout the state to take a fresh look at how mathematics and science education in California might be re-oriented to better meet the goals of these new standards.

The image of a traditional American classroom depicts a teacher standing in front of a large classroom of students lecturing them on a given (and usually very specific) subject or set of problems to complete, while students listen, take notes, and search for the correct answer when prompted. It is an image modeled after two core but sometimes conflicting goals for public education: (1) preparation of students for college and university, and (2) the most efficient way to educate large numbers of students.

Throughout the development of the school system in the 20th century, progressive ideas about the nature of learning competed with the logistics of integrating a largely immigrant population to meet the workforce needs of a growing industrial economy, which required more consistent standards of education for the population at large, reserving postsecondary education for the few.

In contrast, post-World War II America was characterized by the growth of the middle class and a greatly expanded, yet increasingly centralized, public education system. The US began to envision college for more than just the few and put in place a more robust public postsecondary system. This system helped to meet the new demands of an expanding economy, the new role in the world as a global superpower, and global political threats. The launch of Sputnik by the Soviet Union in 1957 also resulted in greater support for mathematics and science education, NASA, and the space race of the 1960’s. In turn, President Dwight Eisenhower initiated the “National Defense Education Act,” which increased funding to education at all levels, with a focus on scientific and technical education (Powell, 2007).

The initial Elementary and Secondary Education Act (ESEA) of 1965, under President Johnson was the first significant infusion of federal funding in public education, which provided extended funding for special education, and additional funding to schools with high concentrations of poverty. This entry of the federal government in public education evolved over the next several decades to support the development of national priorities related to educational reforms, curricular standards, stronger test-based accountability systems, and other centralized functions for a more consistent administration of public education at the federal, state, and local levels.

2 http://www.cde.ca.gov/ta/tg/hs/cahseesuspendfaq.asp


Public Works

Beginning in the 1980’s, a series of reports such as the widely cited, A Nation at Risk (USDOE, 1983), highlighted American teachers’ lack of training to match the kinds of educational preparation needed for success by American youth. Improving professional development for teachers emerged as a primary vehicle for educational reform efforts in the 1990’s and beyond. Other education reforms identified a disconnected, “patchwork” curriculum as the central problem in improving the delivery of education, thus the emergence of standards-based accountability reforms taking hold with greater urgency in the 1990’s and culminating in the test-based emphasis and resulting narrowing of the curriculum of the state and federal reforms of the early 21st Century (Graham, 2013).

As we have entered the second decade of a new century, the school reforms envisioned by proponents of integrated STEM, CCSS-M, and NGSS bring together several strands of what was also envisioned at the end of the twentieth century: (1) embedding research about how students learn from a cognitive standpoint, (2) the importance of motivation and instilling self-direction in the classroom, and (3) the role of the teacher as a facilitator who can be supported through structures for professional collaboration and a more comprehensive approach to continuous professional learning.

Standards Supporting an Integrative STEM Approach “STEM education” today refers to the relatively new reform movement of integrative STEM education (Sanders, 2009). Integrative (or integrated) STEM education involves the explicit incorporation of technology and engineering practices into mathematics and science lessons to organically facilitate interdisciplinary student learning experiences (Tseng, Chang, Lou, & Chen, 2013). For example, while disciplines traditionally operate in isolation of one another, integrative STEM education uses engineering and technology to teach mathematics and science lessons that are synthesized through real-world applicability, integration of technology, problem solving and the like. In implementing an integrative STEM approach, there is both an impetus to better prepare students for the workplace of the future and the notion of educating well-informed citizens who are able to problem solve, communicate, and understand concepts from a variety of viewpoints. Although this has been a challenge, the recent drive to implement common standards may provide a new framework for which integrative STEM might be more widely possible.

Over the past several school years, states across the nation and the District of Columbia have begun to implement the Common Core State Standards in English Language Arts (CCSS-ELA) and Mathematics (CCSS-M)3. A parallel effort by the National Research Council over the past several years has resulted in a new framework for K-12 science education and the Next Generation Science Standards (NGSS), all of which have been adopted in California.

The Common Core standards are designed to have college and career readiness as primary goals. Internationally benchmarked, they send clear signals to students, parents, and educators about the knowledge and skills students need

3 In 2010, California adopted the Common Core State Standards in mathematics and English language arts.

• TheCCSS-MusestheStandardsforMathematical Practices to develop students thinking in the following areas: (1) makes sense of problems and preserve in solving them, (2) reason abstractly and quantitatively (3) construct viable arguments and critique the reasoning of others, (4) model with mathematics, (5) use appropriate tools strategically, (6) attend to precision, (7) look for and make use of structure, (8) look for and express regularity in repeated reasoning.

• TheNGSSidentifieseightpracticesof science and engineering as an essential framework for students: (1) asking questions (for science) and defining problems (for engineering), (2) developing and using models, (3) planning and carrying out investigations, (4) analyzing and interpreting data, (5) using mathematics and computational thinking, (6) constructing explanations and designing solutions, (7) engaging in argument from evidence and (8) obtaining, evaluating, and communicating information.



to learn at each grade level, and to be prepared to enter first-year courses in college without remediation, or to enter workplace training programs for careers that offer competitive salaries. While the standards do not address all of the knowledge and skills students are expected to demonstrate to succeed after high school, they represent academic competencies that are an important step toward school improvement across the nation (Rothman, 2011).

The CCSS-M emphasizes focus and coherence. To achieve coherence, the standards lay out a logical sequence of student learning from grade to grade, intended to lead to college and career readiness by the end of high school. The standards are organized in two parts: (1) a focus on traditional mathematical topics: numbers and operations, algebra, functions, statistics and probability, and geometry, and (2) mathematical practices that “describe varieties of expertise that mathematics educators at all levels should seek to develop in their students. These practices rest on important ‘processes and proficiencies’ with long standing importance in mathematics education” (Rothman, 2011). NGSS is aimed at fostering K-12 students’ deeper understanding of science, in part, by asking them to use the same kinds of practices scientists use. Thus, the standards ask students to apply what they learn through the practices of scientific inquiry and engineering design. They weave together three dimensions: (1) disciplinary core ideas, (2) science and engineering practices, and (3) cross cutting concepts.

Under NGSS, the goal of instruction shifts to explaining phenomenon. This refocusing of goals for learning has real implications for what teachers need to do in helping students develop ideas. This requires teachers and textbooks not simply to present facts and definitions as ends in themselves, but rather, to help students continually work towards explanatory models, developing these ideas from evidence (Leher and Schauble, 2006 as cited in Reiser, 2013). This focus on developing explanations poses challenges for teachers in how to generate lessons through phenomena, and how to identify questions that motivate students (Reiser, 2013).

Similar in many ways to the eight mathematical practices that form the CCSS-M, the NGSS has eight science and engineering practices designed to explicitly incorporate much of what has been thought of as “inquiry.” However, in the NGSS, these eight practices elaborate how to engage in the work of inquiry, and how this work is part of building knowledge by posing questions, designing investigations, building explanations and models of findings, and engaging in argumentation to conduct principled comparisons of competing ideas (Maxwell, 2013). Thus, the eight science and engineering practices in the framework and NGSS emphasize aspects often missing from these earlier interpretations of inquiry in classrooms (NGSS Lead States, 2013).

Teacher Practices Facilitating Cooperative & Experiential LearningAt the heart of an integrated STEM experience for students is the idea that being prepared for the challenges of the future requires strong problem solving skills, teamwork and understanding how to connect theory (or knowledge) with practice (or practical applications). The tensions between traditional or a classical education best suited for those transitioning to an elite college and university system, and the need for a broader, more democratic approach to public education have consistently been at the core of education reform initiatives. This goes back to the late nineteenth century through to the early twenty-first century, especially as learning theory coalesced around the importance of experience in sustaining learning connections. Today, there are parallels in the approach to education embodied by advocates for an integrated STEM approach and the best way to prepare our nation both to be competitive and to achieve equal opportunity.

In the early twentieth century, John Dewey wrote in Experience and Education (1938) about how students and teachers are given limited opportunities to learn through their own experiences in the


Public Works

American education system. Lev Vygotsky introduced a similar idea in the concept of zone of proximal development in the early 1930s, which grew in prominence and was embodied in the reforms of the late 1960’s and 1970’s. The zone is described as the distance between the actual development level or the current knowledge as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance or in collaboration with more capable peers (Vygotsky, 1978).

As experiential learning gained a foothold in the 1980’s, there was a renewed focus on experiences, observations, and social interactions through hands-on activities, group work and cooperative learning. In experiential learning, the teacher’s role is not only to “implant” new ideas, but to also dispose of or modify old ones (Dewey, 1938; Kolb, 1984). Collaboration among students is another aspect of pedagogy that emphasizes facilitation over didactic instruction, and was studied by the National Centre for English Language Teaching and Research in Australia (Burns & de Silva Joyce, 2005). Collaboration embodies joint participation between teachers, whose role is to be knowledgeable and understand how to facilitate assistance in relation to a child’s learning and development. Other research in the cognitive sciences suggests that applied and active learning is a strong environment for most, if not all, students—taking students from being passive listeners to active learners (Grabinger & Dunlap, 1995; Bonwell & Eison, 1991).

Project Based Learning (PBL) is a strategy that combines curriculum with instruction that empowers learners to conduct research, integrate theory and practice, and apply knowledge and skills to develop a viable solution to a defined problem (Savery, 2006). The Buck Institute for Education (BIE) defines PBL as a “teaching method in which students gain knowledge and skills by working for an extended period of time to investigate and respond to a complex question, problem or challenge”(BIE, 2014). Developed by Dr. Harold S. Barrows and his colleagues in the late 1960s, the PBL approach has since been more broadly applied in both K-12 and postsecondary settings, and emphasizes several characteristics that better support how people learn:

• Studentresponsibilityforlearning.• Simulationofproblemsstructuredtoinvolveallstudentsandcollaboration.• Integrationofawiderangeofdisciplinesorsubjects.• Self-directedlearningappliedbacktotheproblemwithreanalysisandresolution.• Aclosinganalysisanddiscussion.• Selfandpeerassessmentatthecompletionofeachproblemandattheendofeverycurricular

unit.• Activitiesvaluedintherealworld.

Another line of research related to effective pedagogical practices is embodied in studies that examine the importance of how student motivation and self-efficacy are supported in classroom settings. As learning theory has developed, and classroom teachers are expected to account for a wider diversity of racial-ethnic, income, language, and other background factors, the study of self-efficacy and motivation has provided support for engaged pedagogical practices. Student learning includes a cycle of inter-related constructs beginning with self-efficacy, which influences goal orientation and motivation.

Mastery-oriented goals can motivate students, thus increasing engagement and self-regulation, and, in turn, increasing student learning and metacognition. This increase in knowledge, skill, and reflection can then raise confidence and self-efficacy. Students are more likely to approach and engage in learning in a manner consistent with mastery goals when they perceive meaningful reasons for engaging in an activity. Tasks that offer personal challenges give students a sense of control over the process or product, or



increase students’ interest over time are the most lasting ways to facilitate learning (Ames, 1992; Schunk, 1984, 1989).

With the school reforms underway due to CCSS-M, CCSS-ELA and NGSS, teachers are being asked to move away from direct instruction, and to move towards using instructional strategies that evoke more elaborate responses or explanations through student inquiry. A point of agreement among mathematics and science educators is that all inquiry involves asking questions and framing explanations (Camins, 2001). For students to be effective problem solvers or critical thinkers, they need to adopt and assimilate the values, attitudes and ways of thinking associated with scientific inquiry and mathematical investigation.

In inquiry-based instruction, students learn, practice and adopt into their everyday existence: questions and conjectures, observations and tools, evidence and explanation, reasoning and proof, communication and critique, and revision and change. These require instructional direction and focus, which means that students need regular opportunities to apply these skills and process them in personally meaningful contexts (Camins, 2001).

CCSS and NGSS both emphasize curricular goals and practices to move from the narrow conception of content knowledge toward an understanding of inquiry that recognizes the importance of social interaction and discourse in developing explanatory ideas (Berland and Hammer, 2012; Duschl, 2008; National Research Council, 2007; Windchitl et al., 2008; all cited in Reiser, 2013; Schoenfeld & Kilpatrick, 2013). Furthermore, the CCSS-M and NGSS practices both highlight argumentation, supported by evidence as a key element in developing explanations and models. They make explicit that the work of building, testing and refining knowledge is realized through discourse. Thus, engaging in discourse and collaborating with others to reach consensus is an explicit element of the practices (Reiser, 2013; Schoenfeld & Kilpatrick, 2013).

As cognitive scientists continue to understand more about the fundamental characteristics of learning, they have come to realize that the traditional classroom’s structures and resources often provide quite poor support for learning, placing students in a passive role. In contrast, technology, when used effectively, can enable ways of teaching that are more directly matched to how children learn, enabling students to: become active learners, work in groups, have frequent interaction and feedback, and connect activities to real world contexts (Roschelle et al., 2000).


Public Works

From Learning Theory to Practice—A New Look for Pedagogy To support specific strategies that blend what we now know about how students learn and the specific classroom structures that support student confidence and self-efficacy, teachers must support instructional strategies that better meet these needs. Palinscar (1998) suggests that while direct instruction can be an effective means of teaching factual content, less evidence exists to support the claim that this type of instruction transfers to higher-order cognitive skills, such as reasoning and problem solving. The effectiveness of direct instruction for equipping students with such higher-order cognitive skills can be improved with scaffolding, strategies designed for English Learners (ELs), communication and discourse, and cooperative learning.

• Scaffolding refers to support that is designed to provide the assistance necessary to enable learners to accomplish tasks and develop understanding that they would not have been able to manage on their own and builds off of Vygotsky’s zone of proximal development. Scaffolding is aimed at enabling students to learn independently, so that teacher support is gradually withdrawn as learners become increasingly able to complete a task alone. Effective scaffolding requires clearly articulated goals, and learning activities that are structured to enable learners to extend their existing level of understanding (Chazan & Ball, 1995).

• Supporting English Learners through SIOP and SDAIE is based on research related to the best instructional practices for students who are second language learners have proven to be more useful when broadly applied in classrooms and for addressing the needs of a growing population of students. Specially Designed Academic Instruction in English (SDAIE) and the Sheltered Instruction Observation Protocol (SIOP) build off of the components of research-based instructional strategies used by teachers to assist ELs with developing knowledge and connecting their background knowledge (Hill & Flynn, 2006).

• Communication and Discourse strategies is based on another line of research suggesting that in reform classrooms, students should talk more and teachers should talk less. To facilitate this new format for classroom dialogue, students are engaged with complex, open-ended problems – in both small groups and as a whole class (Chazan & Ball, 1995). This vein of research likely arises as a result of existing issues with the way many teachers engage their students in dialogue today, as most teachers do not plan or conduct classroom dialogue in ways that encourage the type of discourse that helps students learn. For instance, teachers usually wait less than one second after asking a question for an answer. Yet, as a consequence of the short wait time, the only questions that teachers can ask are those that can be answered quickly and without much student thought (Black, Harrison, Lee, Marshall, & William, 2002).

• Effective questioning is another critically important form of communication and discourse in the classroom. Effective questioning refers to whether or not teachers continue to pose stimulating questions to students as they are engaged in an activity. Asking simple questions such as, “Why do you think that?” or “How might you express that?” can become part of the interactive dynamics of the classroom, and can provide an invaluable opportunity to extend students’ thinking through immediate feedback on their work. After effective questioning is utilized throughout a primary lesson activity, designed followup activities must provide opportunities to further extend student understanding (Black et al., 2002).

• Cooperative learning serves a variety of purposes. Cooperating learning groups may be used to teach specific content, to ensure active cognitive processing of information during lecture or demonstration, and to provide long-term support and assistance for academic progress. Cooperative learning consists of students working together for one-class to several-week durations, to achieve shared learning goals and complete specific tasks and assignments (Johnson & Johnson, 1999).



Programs & Curriculum to Introduce STEM to StudentsWhile STEM is somewhat more established and available to high school students, educators today realize the need for introducing an integrated STEM to students prior to high school. This is especially important for approaches designed to address the underrepresentation of women, youth who live in poverty and historically disadvantaged minority groups to pursue STEM-based higher education (Sanders, 2009). It is difficult for students to learn science with a “traditional” approach through 8th grade and then transition effectively to integrative STEM as soon as they enter high school and prepare for college and the workplace (Mataric, Koenig, & Feil-Seifer, 2007). While there is a growing body of research and substantial investment of funding for the development of these initiatives, there are relatively few that are widely available, especially for elementary students. There are often substantial costs involved in full implementation, including purchasing materials, facilities, and professional development and training.

• Engineering is Elementary from the National Center for Technological Literacy, Museum of Science in Boston offers a Teacher Educator Institute and a “national network of collaborators” to support implementation of its programs and curricula. Engineering is Elementary curriculum is designed for Life, Earth and Space, and Physical science and consists of a materials kit, a teacher guide, and a storybook (EiE, 2014c). Engineering projects and assignments are communicated to students through each curriculum’s storybook, which contain narratives about children from different ethnic backgrounds learning about engineering from adult mentors (EiE, 2014c).

• Lego Mindstorms is a robotics program primarily designed for middle schools to disseminate innovative, STEM-integrated programming. Lego Mindstorms sets include guidelines for teachers to follow, hardware, software and curricula (LEGO Education, 2014). Students might be expected to not only construct a robot and articulate how and why it functions—they might also have to test the effects of various variables on how it operates, while recording their findings in a digital workbook (LEGO Education, 2014). Lego robotics activities are designed as group activities involving project/teamwork as curricular components and have been found to be effective at both deepening students’ understandings of scientific concepts and improving students’ interest in robotics (Lu, Kang, Huang, & Black, 2011).

• Vex Robotics provides materials for middle and high school robotics programs, while leaving programmatic and curricular design and teacher training primarily up to third-party developers. As a result, Vex Robotics provides interested STEM educators and reformers with a variety of curricula to choose from – all of which have been developed by organizations at the forefront of STEM programmatic and curricular design such as: The Carnegie Mellon Robotics Academy, Project Lead the Way, Intellitek, Autodesk, and Analytical Integrated Mathematics (AIM; VEX Robotics, 2012).

• Project Lead the Way (PLTW) is a nonprofit organization that provides innovative STEM-integrated programs and curricula that are aligned with emerging Common Core State and Next Generation Standards to both primary and secondary schools (PLTW, 2014a). PLTW has developed five programs: PLTW Launch (K-5), PLTW Gateway (middle school), and three more specialized high school programs (Engineering, Biomedical Science and Computer Science). PLTW operates through a network of registered schools and districts in which PLTW programs and courses are implemented. In order to begin a program, teachers from registered districts or schools participate in PLTW professional development that includes: Readiness Training, Core Training, Ongoing Training, and Professional Learning Communities. Furthermore, PLTW provides school support and technical assistance and has an option for program certification for its high school programs to continue to build consistency in implementation.


Public Works

School Structures and Other Initiatives to Expand Access to STEMProgrammatic and curricular approaches for integrative STEM are more fully developed at the high school level, in part because of efforts to reform traditional, comprehensive high school education and to improve career-technical education to better align with a new more global and technologically competitive economy. While not specifically designed to support STEM, there are many examples of high school approaches that offer opportunities for introducing students to integrative STEM in a structured way. However, to make these opportunities more broadly available, those interested in expanding access to STEM must address a history of sporadic implementation and the tremendous variation that exists from school to school.

• Inanefforttoreinvigoratehighschoolcareertechnicaleducation,communitycollegeandotherpostsecondary connections, in July 2014, California created the California Career Pathways Trust providing $250 million in funding to school districts, county offices of education, charter schools, regional occupational centers or programs and community college districts for a competitive grant program to fund career pathways and collaborations among business, K-12 and postsecondary education and community organizations (http://www.cde.ca.gov/ci/ct/pt/). The first set of awards was announced in May 2014.

• AnotherinitiativeinCaliforniabuildingonrecentinterestinSTEMpreparedness—theCalifornia STEM Learning Network (CSLNet) is a non-profit organization that is providing technical assistance and tools to build the capacity of regional networks throughout the state (CSLNet, 2014). With a mission to prepare the nation’s most STEM-capable graduates by coordinating and activating a statewide network representing all STEM stakeholders, CSLNet’s vision is for all students to graduate with the STEM knowledge and skills required for success in education, work and their daily lives. Without support for particular programs as such, CSLNet is instead convening the broader community and has hosted STEM summits and other opportunities for stakeholders to learn about and build STEM opportunities.

• Career Academies, the National Academy Foundation (NAF) and the California Partnership Academies are one of the most systematically studied approaches to downsizing the learning experience of high school students within a larger school environment. Under the academy model, high schools organize the curricula and education program for a subset of students (usually about 300) with a team of teachers who organize integrated curriculum around one or more themes, typically career or occupationally related. NAF is a network of partnerships that began 30 years ago between business partners and schools to provide traditionally underserved high school students with access to quality careers (NAF, 2014). While NAF has programs designed for five different career “themes,” three of these themes are directly related to STEM disciplines, including information technology, engineering, and health sciences.

• TheLinked Learning Alliance is a statewide coalition of education, industry, and community organizations first established in May 2008 to expand access to an approach to high school integrating academics with real-world learning opportunities in a range of industry sectors. Linked Learning has received support and is expanding throughout the state of California.

• STEM-integrated Smaller Learning Communities or SLCs are part of an effort to improve student outcomes that began in 2000 with federal funding to downsize comprehensive high schools with more than 1,000 students (SLCs; USDOE, 2008). These programs are more likely to offer a broader and more in-depth STEM curriculum, expanded AP classes, higher levels of student-teacher interaction, and have school-university or school-corporation partnerships (Atkinson & Mayo, 2010).

• Magnet schools began as a protest against racially segregated schools in the 1960’s due to growing concerns for academic innovation and achievement, especially in poor urban communities. While the SLC movement was based on a recognition that large, comprehensive high schools were severely underperforming, magnet schools were an educational reform strategy designed to address and alleviate educational inequality (Siegel-Hawley & Frankenberg, 2012; Waldrip, 2013). Because of their similar structures, goals, and roles within their districts, the strengths of STEM-integrated magnet schools overlap with the strengths of SLCs (Siegel-Hawley & Frankenberg, 2012).



From Teacher Training to Professional LearningAcross the United States, educational reform efforts are dramatically raising expectations for students and teachers. To meet these new expectations, teachers will need to deepen their content knowledge, and learn new methods of teaching. As a result, teachers also require more time to work with colleagues to critically examine the new standards being proposed, and to revise their curricula. Teachers will also need opportunities to develop, master and reflect on new approaches to working with children. Over the past twelve years, the CaMSP initiative has supported a range of innovative professional development, much of which embodies the following concepts.

In past reform efforts, professional development has been characterized almost exclusively in terms of formal education activities, such as different courses or workshops offered several times a year (Corcoran, 1995). These were predominately one-time workshops that focused mostly on awareness or general knowledge rather than specific skills, or models that have little basis in what is known about effective curricula and classroom instruction (Pianta, 2011). High quality professional development is now a central component in nearly every modern proposal for improving education. For example, California’s Superintendent’s Quality Professional Learning Standards (QPLS; California Department of Education, 2013) focused on developing standards that are now the cornerstone of quality professional learning and include elements that cut across specific content knowledge, pedagogical skills and dispositions.

High quality professional development focuses on several characteristics that affect classroom learning and teaching: 1) high quality professional development must immerse participants in inquiry, questioning, and experimentation, and therefore model inquiry forms of teaching; 2) professional development must be both intensive and sustained; 3) staff development must engage teachers in concrete teaching tasks, and be based on teachers’ experiences with students; 4) professional development must focus on subject-matter knowledge and deepen teachers’ content skills; 5) professional development must be grounded in a common set of professional development standards, and show teachers how to connect their work to specific standards for student performance; and 6) reform strategies must be connected to other aspects of school change (Supovitz & Turner, 2000).

Professional development programs are increasingly designed to be systematic about bringing about change related to the classroom practices of teachers, in their attitudes and beliefs, and in the learning outcomes of their students (Guskey, 2002). Recent research suggests that effective professional development is intensive, ongoing and connected to practice that focuses on the teaching and learning of specific initiatives, as well as building strong working relationships among teachers (Darling-Hammond & Richardson, 2009; Darling-Hammond, Wei, Andree, Richardson, & Orphanos, 2009; Desimone, Porter, Garet, Yoon, & Birman, 2002; Guskey, 2002). Collective work in trusting environments provides a basis for inquiry and reflection, and allows teachers to raise issues, take risks, and address dilemmas in their own practices.

Research suggests that professional development is most effective when it addresses the concrete, everyday challenges involved in teaching and learning specific academic subject matter (Darling-Hammond & Richardson, 2009). Similarly, the National Council for Teachers of Mathematics recommends that teachers pose meaningful, complex tasks for their students, provide opportunities for students to engage in real world problems, and use manipulatives and technology (Sparks & Loucks-Horsley, 1989).

Teacher Leadership and Supporting Healthy Learning EnvironmentsAlthough states have maintained a focus on recruiting and retaining teachers, many schools and districts still face daunting challenges in ensuring qualified and competent teachers (Berry & Hirsch, 2005). Teacher retention is particularly difficult for schools considered hard to staff – such as those with high concentrations of low performing, low income students. Those schools experience high teacher turnover


Public Works

and relatively high percentages of teachers who are less than fully certified. High teacher attrition is expensive, and undermines long-term improvement in high-needs schools. Furthermore, research indicates that professional preparation directly impacts student achievement, as does the retention of more experienced and credentialed teachers.

Studies show that positive relationships with colleagues and principals counteract the primary causes of teacher attrition, which are unhealthy teaching and learning environments, inadequate support systems and bureaucratic impediments. Thus, while PLCs, mentoring and induction programs might provide schools with opportunities to foster positive relationships amongst teachers and staff, programs such as these are only effective if they are made up by a network of healthy relationships (Berry, 2010). Strong and supportive school leadership, time for teachers to develop their teaching craft, and sufficient materials and resources to teach effectively have been found to improve teacher working conditions (Berry & Hirsch, 2005; Futernick, 2007).

While the typical career ladder for a teacher interested in career advancement has traditionally meant becoming an administrator, Harrison and Killion (2007) found that teacher leaders can assume a wide range of roles to support school and student success without necessarily entering the administrative track. Whether these roles are assigned formally or shared informally, they build the entire school’s capacity to improve.

Teacher leaders can assume a variety of roles including sharing of instructional resources to support their colleagues, instructional specialists supporting specific content areas and school leadership in the form of lesson planning or grade level teams. When teachers are encouraged to approach one another for resources and research, they are more likely conduct research in their own classroom, which promotes a school culture of accountability and professionalism (Corcoran, 1995). Teachers who aim to be school leaders on their campuses share the visions of their schools, align their professional goals with those of their schools and districts, and assume responsibility for the success of their schools as a whole (Harrison & Killion, 2007).

The Road Ahead for STEM Teaching and LearningThe emergence of STEM education reform recently has been based on the perceived need to increase our educational system’s capacity to prepare more qualified workers and students for STEM careers and undergraduate programs. However, strategies for “fitting” STEM within the existing structures of education continues to be a challenge. At the high school level, accountability efforts and increasingly rigid course requirements for college admission make the broad adoption of integrative STEM difficult. If the goal is to prepare more qualified STEM undergraduate students, then educators must continue to build on existing structures and work toward implementing high school science and mathematics courses that provide both integration and academically relevant content specialization, both of which are possible with the new areas of emphasis embodied in CCSS-M and NGSS. While a focus on student achievement and equity leads educators to prioritize the need to improve critical thinking skills and specific content skills identified in new mathematics and science standards, increasing student interest in STEM is a second essential component to addressing America’s STEM needs. Incorporating integrative STEM into elementary and middle schools is a crucial next step for STEM reformers, especially those interested in more effective ways to make STEM career accessible to traditionally underserved minority groups, young women and low income students.

Teacher leaders can:• Providedemonstrationlessons.

• Implementnewideasthroughco-teaching or observing and giving feedback.

• FacilitatePLCsorprovideprofessional development about adopted curriculum, pacing and developing shared assessments.

x2 • y2 = z2

F=ma

Al2(SO4

)3

79

Au197.0

F = Gm1 m

2 /r 2

A = πr2

Section 4:

Evaluation Results

Overview


Public Works

Section 4: Evaluation Results OverviewThis section of the report incorporates data collected in 2014-15 from Cohort 10 CaMSP partnerships and is organized as a summary of implementation across the five key features of CaMSP. The PW evaluation uses the following implementation rubric as its analysis framework:

Key Features Rubric Scale

Partnership-Driven: The partnership represents its target population of districts, teachers and institutions of higher education. All partners exhibit a high level of commitment to the partnership. The governance structures include both the trainers/IHE and the districts as equal partners in planning the curriculum and logistics. The target population of the grant is served.

Teacher Quality: The partnership has created a cohort of teachers enrolled in all aspects of the professional development. They are on their way to meeting the target number of teachers involved and the hours required. The approach to professional development is tied to state standards and is focused on both improving the content knowledge and pedagogical approach of teachers. The professional development is research-based, high quality for both intensive training and followup components that include monitoring of implementation.

Challenging Courses & Curricula: Professional development is aligned to standards and aimed at transforming and improving instruction. The project is creating new challenging courses, lessons and curricula for pre-service or existing teachers and/or students. New courses, curricula, expectations or experiences for students will result from the professional development that teachers receive in this grant.

Institutional Change & Sustainability: The role of the IHE is clear and integral to the project. The IHE is involved in the planning, curriculum development and delivery. The education department and the discipline department (mathematics/science) are involved. Teachers can receive credit for their professional development. The institutions (IHE and Districts) are impacted by the project in terms of a tangible result. There are sustainable elements of the grant.

Evidence-based Design & Outcomes: Partnership uses a research or evidence-based model for professional development. The evaluation plan makes sense for the project. The design and measurement system will produce an impact on teacher and classroom quality. There is a cohort of teachers for which examining the impact on student outcomes makes sense. A pre-/post-assessment of teacher knowledge is conducted and local student assessment is conducted (above and beyond CSTs).

High-level implementation

No evidence of implementation

About the PartnershipsFor the CaMSP program, the 20 partnerships funded under Cohort 10 represented a substantial shift in focus of CaMSP to an integrated STEM approach, where partnerships were directed to design professional development models for grades K-12 that integrated the disciplines of science, technology, engineering and mathematics (STEM) through a variety of models for professional development. Cohort 10 partnerships had to choose at least one core discipline (mathematics and/or science) and at least

Evaluation Results Overview


one supporting discipline (engineering and/or technology) to provide content instruction, pedagogical strategies and curriculum development support to participating teachers. In addition, Cohort 10 partnerships were directed to support implementation of California’s standards under the Common Core State Standards (CCSS-M) and the Next Generation Science Standards (NGSS).1

Cohort 10 varies substantially in the number of districts participating in the grant, with three of the partnerships involving more than ten school districts (Table 4.1). This cohort is a mix of multi-district (12) and single-district partnerships (8), with seven of the multi-district partnerships identifying a county office of education (COE) as the lead LEA. The number of targeted teachers ranged from 45 to 100 with a total of 1,110 teachers targeted for professional development support.

Under Cohort 10, the grade levels of participating teachers was expanded from grade 3 to Algebra I for mathematics and grades 3 to 8 for science to at least one of the following grade spans: K-2, 3-5, 6-8, and 9-12 or contiguous grades (K-6, 3-8, 6-12). Twelve partnerships are focusing on the primary and lower secondary grades and eight partnerships are focusing strictly on secondary grade levels.

Cohort 10 was designed to incorporate an integrated approach to professional development that included the disciplines of science, mathematics, engineering and technology. Eight of the twenty partnerships identified themselves as fully-STEM integrated, incorporating all four disciplines. The others selected a subset of the four. In terms of focus on mathematics and/or science, the majority of Cohort 10 partnerships have identified both mathematics and science as the core discipline (11). Five partnerships identified mathematics only and four identified science only. For the supporting discipline, six partnerships identified technology, three partnerships identified engineering, and eleven partnerships identified both technology and engineering (Table 4.1).

1 The California Board of Education adopted the Common Core State Standards in Mathematics (CCSS-M). Next Generation Science Standards or NGSS were adopted in California in 2013.


Public Works

Table 4.1: Cohort 10 Partnerships by Lead LEA

Lead LEA # of Districts

Grades Targeted

# of Teachers

Core Discipline

Supporting Discipline

ABC USD 1 7-8, 9-12 51 Science Engineering

Anaheim UHSD 1 7-8, 9-12 52 ScienceTechnology and

Engineering

Butte COE 6 K-6 60Science/

MathematicsTechnology and

Engineering

Coachella Valley USD 3 6-8, 9-12 50 Science Engineering

Escondido USD 1 6-8 50Science/

MathematicsTechnology

Fortuna ESD 1 6-8, 9-12 45Science/


Engineering

Hacienda La Puente USD 1 3-5 50 Mathematics Technology

Lakeside USD 2K-5

50Science/


Lamont ESD 3 3-5, 6-8 47 MathematicsTechnology and

Engineering

Orange COE 7 3-5, 6-8 63Science/


Engineering

Paso Robles Joint USD 6 3-5 50Science/


Engineering

Rialto USD 2 3-5, 6-8 66 Science Technology

Sacramento COE 4 6-8, 9-12 50 Mathematics Engineering

Salinas City ESD 1 K-2, 3-5 100 Mathematics Technology

San Joaquin COE 6 9-12 50Science/


San Rafael City ESD 4 3-5, 6-8 50Science/


Engineering

Shasta COE 35 6-8, 9-12 60Science/


Engineering

Tuolumne COE 11 K-2, 3-5, 6 55Science/


Engineering

West Contra Costa 1 3-5 50Science/


Engineering

Yolo COE 13 6-8, 9-12 61 MathematicsTechnology and

Engineering

Source: Project Applications



Findings for Key Feature 1: Partnership-DrivenCore PartnersIn general, the partnerships across the 20 Cohort 10 sites are strong and committed to developing what they proposed for the RFA. There are a variety of perspectives related to integration of STEM content areas, with some partnerships identifying specific strategies for the rollout and addition of content areas over time and others attempting to integrate across all disciplines from the very beginning of implementation. While there are a substantial number of partnerships that have collaborated in past CaMSP cohorts, there are about a quarter to a third of the partnerships that are new to CaMSP and/or have invited new professors or collaboration from IHE partners, especially related to engineering and technology.

The majority of Cohort 10 partnerships are working across multiple districts. In these partnerships, participating districts stay informed about the grant activities through the quarterly core partnership meetings. Many of the partnerships are led by a county office (COE) or one is included as a partner LEA. COE resources supporting these partnerships include grant communications/dissemination of

grant activities through emails, meetings with partners, and other strategies for dissemination. One COE is also involved with one partnership as a professional development provider.

Among the partnerships that have previously collaborated (some as far back as CaMSP Cohorts 1, 2 and 3), these strong working relationships have had a positive effect on the ability of partners to plan and implement quality professional development, even as they plan new and/or untested approaches to integration across content areas. Often, partners are taking a previous approach in one content area and adapting it over time to include engineering problems and/or technology tools or solutions. When meeting with the partnerships, there is a palpable sense of purpose and strong, productive working relationships.

This cohort of partnerships also has a large number of partnerships that are led by a COE (7 of 20) and involve a wide range of districts. Several of these COE’s have had previous experience with CaMSP (in some cases managing partnerships) and are expanding on those initiatives through Cohort 10. For these partnerships, the integration of curriculum across disciplines is new and an adaptation of previous initiatives. In a few cases, COE’s are new to managing CaMSP projects and are partnering with new districts. There has been a learning curve related to recruitment

and retention of teachers, understanding the scope of CaMSP requirements and establishing realistic expectations related to delivering and disseminating curriculum.

The majority of the core partners (IHE, school administrators and professional development providers) continue to work together to create the activities for the summer institutes and classroom followups adapting to needs of participants and making adjustments as needed. One partnership not only has its core partners involved in implementing and planning grant activities, but a professional design team that

Cohort 10 University Partners: Cal Poly PomonaCal Poly San Luis ObispoChapman University (MESA Office) Columbia College CSU BakersfieldCSU ChicoCSU East BayCSU FullertonCSU Long BeachCSU Monterey BayCSU San Bernardino CSU San MarcosHumboldt State UniversitySacramento State UniversitySan Diego State UniversitySan Francisco State UniversityUC DavisUC IrvineUC RiversideUniversity of the PacificWhittier College

Top 3 Ways CaMSP Supports Student Learning for Core Partners1. Instructional strategies/

pedagogy (94%)2. Content knowledge (92%)3. Instructional strategies to

support STEM (89%)

Source: 2014 Core Partner Survey (n=130)


Public Works

consists of the IHE professors, teacher leaders from a previous CaMSP grant, educational technology experts, high school science teachers, and CTE teachers to provide expertise on the content focus.

Evolving Role of the Regional PartnersThe Cohort 10 RFA outlined a partnership structure that blended the idea of core partners with outreach to additional partners, primarily to support real-world integration in the workplace and nearby districts and schools in which to have participating teachers disseminate their work through the idea of a Regional Collaborative. In contrast to the strength of the core partnerships, in the initial visit, the role of the Regional Collaborative Partnership (RCP) was fairly undefined at most, if not all, Cohort 10 partnerships. However, the RCP was intended to provide support to teachers and students as the

professional development model unfolded and was refined over time. There were a range of types of partners that had been recruited for the RCP from employers and chambers of commerce to other county offices of education and school districts to non-profits and government agencies. In a few partnerships, the Regional Collaborative remained fairly limited and presented an opportunity for further definition and possible recruitment of additional partners.

By and large, in interviews of regional partners, there was a great willingness to support the partnership even though that support was yet to be defined. Most partnerships intended the RCP to eventually provide application to mathematics and science through

job shadowing, teacher externships, field trips and the like. In general, there was a better connection to Regional Collaborative Partners in partnerships at the secondary level in which the immediate connection to postsecondary and careers may be more apparent. For those partnerships working with teachers in kindergarten to 5th grade, there was more emphasis on the professional development and integration of new standards first before involving and/or planning the role of the RCP. While there is some discussion related to organizations and initiatives that focus on attracting students to STEM fields and higher education (such as MESA) in a few partnerships, it remained unclear in the beginning of implementation how these organizations would be supporting teachers professional development.2

In Fall 2014, 78 of the 148 regional partners responded to a survey regarding their role in the partnership (52% response rate). Results from the survey indicate that the RCP is bringing in new partners and involvement in CaMSP—with more than three-quarters (77%) indicating they serve only on the RCP, not the core partnership. In addition, the RCP is bringing in both non-profit organizations (15%) and businesses or employers (14%) in support roles for CaMSP teachers. Regional partners expressed support for the partnerships in which they were involved, with many indicating that they had helped in the initial planning and implementation and were looking forward to more clarification about how they could directly support teachers and students.

2 Mathematics, Engineering, Science Achievement (MESA) engages educationally disadvantaged students so they excel in mathematics and science and graduate with math-based degrees. MESA partners with all segments of California higher education as well as K-12 institutions.

Top 3 Ways CaMSP Supports Student Learning for Regional Partners1. Interest in STEM careers

(92%)2. Interest in mathematics/science

(87%)3. Interest in STEM college

courses/majors (86%)

Source: 2014 RCP Survey (n=78)



Figure 4.1: Membership in the Regional Collaborative Partnership (n=78)

Source: 2014 RCP Survey

The majority of the regional collaborative partners provided partnerships with resources and assistance in planning content lessons or classroom activities and job shadowing experience for teachers and students. Regional partners in several partnerships also provided externships for teachers in pathways or academies and developed field trips for teachers and students in STEM careers. One project director mentioned that a regional partner provided trainings, camps and workshops for students, support for afterschool engineering and science programs, and has provided coaches and interns for schools who do not have an onsite coach. Another partnership’s regional partner provides districts with career pathways in engineering. One RCP has focused on training and workforce development in the alternative energy sector where students can attend this pathway and receive training and certification. Partnership directors reported that many regional partners have become mentors to teachers to help and assist them in their area of expertise. Partnerships planned to continue to expand the role of the RCP in second and third year of implementation.

Building off Existing and Connecting to New PartnersThe majority of partnerships have been working with the local business community to help improve student interest in science and mathematics and to provide teachers with externships, mentorships or job shadowing. The majority of these community relationships had existed before the grant, but were being fine-tuned to support CaMSP and local goals. The partnerships are using these connections to build STEM Centers or are putting a stronger emphasis on STEM labs and academies in the districts with the help of site and district administrators and the local business community. Partnerships that included a county office indicated that relationships to community partners have been strengthened due to the grant. Several partnerships reported receiving additional funding from outside agencies to continue and build on implementation within the districts. For example, one partnership received a grant from Boeing to start implementing engineering instruction to all elementary schools within their district.

Examples of Regional Partners Roles• Collaborationeventwhere

exemplary teacher projects and masters will be presented.

• AssistwithSTEMclassprojects,guest speakers and fieldtrips for students.

• Providetours,mentoringandprofessional development for teachers.

• ParticipateinSTEMOlympicDay and Science Fairs.

• Participateinjobshadows,externships and practice seminars.

• Disseminationoftheprojecttoother businesses and industries.

• Communityapplicationstocurriculum (local community issues).

• Produceandhousebestpracticevideos.

28%

20%

18%

15%

14%

5%

0% 10% 20% 30%

Postsecondary Education Institution

K-12 Institution

County Office of Education

Non-profit or CBO

Business or employer

Other


Public Works

Site and District Administrator Involvement Another aspect of partnerships is to involve site administrators, either actively as a separate component of outreach and management or through their implicit support of teacher participation by facilitating and assisting the participation of teachers and ensuring that site-based priorities are not in conflict with the goals of CaMSP. For Cohort 10 partnerships, the level of participation of site or district administrators reported by project directors varied from non-participation to active participation. Many of the projects had district and some site administrators involved as either members of the core partnership and/or were on the design/planning team. These administrators attended the core and regional meetings and if they were unable to attend they were kept abreast through communication from the project director. In many partnerships, site or district administrators that are actively involved in the grant have attended or were expected to attend an all-day professional development to become familiar with the CaMSP project and goals. Some partnerships had professional development training sessions, specific principal trainings to review the activities of their teachers’ professional development, or a training that focused on the district approach to mathematics and science implementation. Administrators in one partnership visited the coaches working with participating teachers and another partnership invited their literacy and ELD coaches to the trainings.



Findings for Key Feature 2: Teacher QualityAs part of its evaluation, PW works with each project to understand the underlying goals and model for delivering a minimum of 60 hours of intensive professional development and 24 hours of classroom followup. While the intensive hours are designed to support new learning for teachers about content, pedagogy and how these ideas interact, the classroom followup component is designed to support teachers to practice and plan how to use what they have learned through the intensive professional development. The three most common approaches include: coaching, lesson study and professional learning communities or PLCs, which form a continuum of support and professional collaboration.

Cohort 10 partnerships designed one- or two-week summer institutes and other workshop days during the school year to fulfill the 60 intensive hours required for each teacher. While most summer institutes were designed to be one week in duration, there is some range in terms of length. A few have most or all of their intensive hours during the summer with a few partnerships experimenting with delivering days of professional development during the summer in multiple segments. While some identified specific topics for their institutes from the mathematics or science standards, most were thematic including: medical and energy sectors, rollercoasters, weather principles, ecosystems, gardening and plant growth, fireworks, forensic medicine, waterways and water treatment, and others. Quite a few of the projects had an environmental or ecology lens that addressed topics that could be embedded in standards at multiple grade levels.

For classroom followup strategies, more than half of Cohort 10 partnerships have included lesson study but most blended several different strategies for school year support of participating teachers. Seven partnerships have workshops days, six have coaching that either compliments the workshop days or lesson study or professional learning communities (PLC). Only four partnerships identified PLC’s as a followup classroom strategy and two indicated teachers would participate in externships at some point during training, though there are several more that are incorporating field days into their summer or school year intensive hours and have indicated that teachers may possibly participate in work-based learning type activities as the project progresses (Table 4.2).

Approaches to Classroom Followup in CaMSP

Coaching emphasizes supporting individual teachers in their classrooms.

Lesson study involves the facilitation of teams of teachers to improve practice based on an action-research model first developed in Japan.

PLCs can be a district strategy often organized school-wide or by grade or subject area, whose aim is to examine and understand student data and embed ongoing professional development in the organization and decision-making for groups of teachers.


Public Works

Table 4.2: Classroom Followup Models and Teacher Support

Lead LEA

Discipline Focus(STEM)

Classroom Followup Models

Workshop Days

Lesson Study Coaching PLC Externship

ABC USD SE x

Anaheim UHSD STE x

Butte COE STEM x x

Coachella Valley USD SE x

Escondido USD SMT x

Fortuna ESD STEM x x

Hacienda La Puente USD MT x x

Lakeside USD STM x

Lamont ESD MTE x x x

Orange COE STEM

Paso Robles Joint USD STEM x

Rialto USD ST x

Sacramento COE ME x x x

Salinas City ESD MT x x

San Joaquin COE SMT x x

San Rafael City ESD STEM x

Shasta COE STEM x

Tuolumne COE STEM x x

West Contra Costa STEM x x

Yolo COE MTE x

Source: Project Applications and Local Evaluation Plans

Integration of Engineering and TechnologyWhile professional development throughout Cohort 10 has identified elements of the mathematics and science standards depending on the core discipline or disciplines selected, there is little definition yet with regard to technology in terms of standards and, in some cases, approach to integration. In addition, the integration of engineering was also a new aspect of the implementation of STEM under Cohort 10 including making brand new connections to engineering departments on IHE partner campuses and embedding a basic understanding of the Engineering Practices outlined in the NGSS.

In partnerships that selected multiple core and supporting disciplines (STEM), most have referred to mathematical practices and/or NGSS engineering practices as the basis for content and instructional integration. In terms of identifying specific components of the new standards adopted in California in both mathematics and science, partnerships leading with (or having professional development experiences from) a single core discipline tend to be building professional development topics from either a mathematics or science perspective, and then blend or integrate ideas from the supporting disciplines such as problem or project-based learning (the Buck Institute approach to project-based learning was identified most commonly). In a few cases, this integration is occurring at the end of the first year and into the second and third years of implementation. In addition to CCSS-M and/or NGSS, about half of the partnerships identified a focus on strategies for the success of English Learners and/or integration on Common Core Language Arts standards into professional development.

The technology component of many partnerships varies widely. Technology approaches include teachers



learning how to use Google documents; creating, editing, and posting video; and developing multi-media presentations. Projects are also asking participants to use “hangouts” or other online collaboration tools to design, collaborate and store units; use iPads and Chromebooks in the classroom for student curriculum such as data and graphing and to film mathematics situations. One project has developed an online badging system to track teacher progress through various components of the professional development. There are only a few projects focused on actual computer science, robotics or app creation.

As the projects have progressed, however, and project directors and partners have a better understanding of what participating teachers might need in terms of support, these approaches have been refined and become better defined. The focus continues to be more about the use of technology in the classroom rather than a specific approach embedding computer science, coding or robotics. Most projects have viewed this aspect of implementation with the most flexibility and have been creative about how to incorporate it without overwhelming participants in terms of implementation in the classroom.

Embedded Support for Teachers in the Classroom and Leadership DevelopmentIn terms of the collaboration components designed for participating teachers, there is a strong focus on pedagogy including peer critiques, inquiry investigations, experimental designs, engineering product design cycles, literacy activities, problem-based learning, scientific presentations and exhibitions, and productive struggle time. Some partnerships are developing vertical teams (articulated across school levels), others horizontal (grade level) teams. This aspect of implementation has been reported as the

most fluid and there are adjustments to the topics, logistics or other aspects of how they are fulfilled. For example, while only a few identified externships for teachers in their original proposals, many partnerships reported that teachers had work-based or other community-

based experiences based on interactions with RCP members and opportunities that arose during the first year of implementation.

Most partnerships are building off of previous experience with more than half of partnerships focused on lesson study. Several partnerships are working with WestEd’s K-12 Alliance to implement lesson study (See side bar to right).3 A few are using coaching, with some blending coaching and lesson study. Very few have identified professional learning communities as their classroom followup approach. While many have identified lesson study, in the beginning only a few could articulate how lesson study would be adapted for the products or curriculum that was

3 More information about WestEd’s Teaching Learning Collaborative or TLC can be found at www.wested.org.

The lesson study gave me an experience I never had before...to try a new lesson out and immediately make changes to it and teach it again. That cannot be done in a typical day...and hope it will work so I loved being able to try something out and hash it out with other people that were observing, ask how we can make it better, and an hour later try it again.

—CaMSP Participant

West Ed’s K12 Alliance is a partner in several CaMSP Cohort 10 partnerships where it has adapted the Teaching-Learning Collaborative (TLC), a form of lesson study developed by the K12 Alliance.

Small teams of teachers work collaboratively over the school year with a WestEd/K12 Alliance facilitator to design a quality lesson; to collaboratively teach the lesson, debrief, and refine the lesson; then to re-teach and refine the lesson again.

West Ed facilitators also train leaders to use the TLC model to design and refine teaching in their own sites. Leaders are trained in the TLC model, shadow a WestEd facilitator onsite with teachers, and then conduct a coaching session themselves and get feedback from experienced facilitators.

TLC includes the 5E model of instructional design, developing conceptual flows for student understanding, specific use of questioning strategies, student discourse and assessment strategies to capture student understanding.


Public Works

envisioned (units and/or modules). In addition, in terms of integration across disciplines, the traditional lesson study approach of developing and refining a single lesson focused on a single topic continues to be adapted for STEM and integrated learning in order to make sure that partnerships are on track with developing the curriculum and/or approach identified in the original proposal.

Leadership development is also being defined as the partnerships implement their projects with an eye to dissemination in the third year of implementation. Many partnerships are training a cadre of facilitators or others to deliver content and conduct lesson studies as well as math, science and technology coaches. In addition, partnerships are exploring different uses of online technology and video or multi-media strategies to develop online courses for teachers, methods for collaboration and other ways to engage teachers in the process (ex. EdPuzzle, Schoology, YouTube, blogs). These strategies are also being explored for make-up of professional development hours.

Teacher Recruitment and RetentionA variety of strategies were used to recruit teachers who were identified in the original proposal. In addition, partnerships continued to recruit in the spring and summer as the first summer institute was finalized in order to meet their targeted number of teachers. Recruitment strategies included emails, outreach to principals, department chair meetings, site visits and group recruitment meetings. In one case, individual letters were sent to each teacher in the district. Most partnerships did not have a challenge in recruiting the target number of teachers for the summer institute, but in a few cases, there was some teacher attrition. Some partnerships are convening meetings with principals to foster administrative support for the professional development in order to secure retention and provide classroom support for the training. One retention strategy was the recruitment of teacher teams, consisting of both mathematics and science teachers from the same schools, who work together in creating modules that will address both the mathematics and science standards throughout the project.

Each year, teachers participating in CaMSP are surveyed to find out about their overall satisfaction with the quality of the professional development. In spring 2015, 1,113 teachers (93% of participants) responded to the survey. In general, survey responses and focus groups conducted in each of the Cohort 10 partnerships indicate very high levels of satisfaction with CaMSP programming despite the wide range of approaches that exist and the variation in content focus and strategy from one partnership to the next. A range of 81% to 93% of teachers reported they were satisfied or very satisfied with various dimensions of CaMSP training at the end of their first year of participation (Figure 4.2).



Figure 4.2. Satisfaction with the quality of the professional development (% Satisfied and Very Satisfied) (n=1,113)

Source: 2015 Teacher Survey

In the 2015 survey, teachers were also asked about the extent that CaMSP was helping them professionally. Teachers indicated that the professional development was supporting them in a number of ways—most importantly with instruction and pedagogy (88%), hands-on learning (84%) and understanding modeling and real world applications in teaching (83%). Slightly fewer, but still a large majority of the participating teachers, indicated that CaMSP was increasing their content knowledge (81%), aligning instruction to new mathematics (75%) or science (68%) standards and implementing project-based learning (72%).

Teachers report that participation in CaMSP professional development can be intense at times, with a three year commitment that includes followup and assignments during the school year. However, almost universally, teachers enthusiastically embraced the challenges posed by the professional development programming—mentioning how the training supported the type of pedagogy it encouraged and noticing excitement from their students about new ways to learn and model what scientists, mathematicians and engineers do. Many teachers observed that the professional development evolved over time, as the professional development teams solidified what they were doing and what they were asking participating teachers to do, which increased their own enthusiasm for teaching.

When asked about the opportunities provided by CaMSP, many teachers indicated that this was the only place in which they were learning about integrated STEM and how to engage in the learning and pedagogical emphasis on engineering practices and modeling in new mathematics and science standards.

93%

88%

88%

85%

85%

85%

82%

82%

81%

70% 80% 90% 100%

Quality of the trainers

PD content

Overall rating of PD

Pedagogy or instructional methods

Quality of summer activities

Quality of school year activities

Quality of the coaching

Impact on teaching

Focus on aligning with standards

Top 3 Ways CaMSP Supports Teachers1. Instructional strategies/

pedagogy (88%)2. Importance of hands-on

learning (84%)3. Understand modeling

and real world applications in teaching (83%)

Source: 2015 Teacher Survey (n=1,113)


Public Works

Findings for Key Feature 3: Challenging Courses & CurriculaCohort 10 partnerships are required to produce STEM products or curricula and disseminate them within their districts, schools and regions over the course of the three years of implementation. Table 4.3 provides a summary of what partnerships have indicated they are developing based primarily on information provided in the proposal and through verification in interviews conducted during site visits. As the products are being developed, the terms are becoming more defined. One aspect of the evaluation is to collect information regarding the number of products that will be produced, who is developing them (individual teachers vs. teacher teams vs. PD provider models) and how the terms are defined. For example, in some cases, online classes are proposed as both a product and a professional development strategy (intensive and/or classroom followup). Nearly all partnerships plan to either display or collect products in an online format. Many have suggested they will be made available to non-participating districts through Web-based portals and other strategies for dissemination. Partnerships have also indicated they will design Websites to share information among participants and to a broader community.

Table 4.3: Curriculum Products

Lead LEA

Discipline Focus(STEM)

Curriculum Products

Writing Lessons

Online Classes

PBL* Lessons

Modules/Units Lessons Website

ABC USD SE x x x

Anaheim UHSD STE x x

Butte COE STEM x x

Coachella Valley USD SE x x

Escondido USD SMT x x x x

Fortuna ESD STEM x x

Hacienda La Puente USD MT x x

Lakeside USD STM x x x

Lamont ESD MTE x x x x

Orange COE STEM x x x

Paso Robles Joint USD STEM x x x

Rialto USD ST x x x

Sacramento COE ME x x

Salinas City ESD MT x x

San Joaquin COE SMT x x

San Rafael City ESD STEM x x x

Shasta COE STEM x x x

Tuolumne COE STEM x x x

West Contra Costa STEM x x x

Yolo COE MTE x x x

*PBL refers to Project Based LearningSource: Project applications, 2014 site visits and 2015 phone interviews

Partnerships are creating new single-disciplinary, interdisciplinary, and multidisciplinary curriculum for students. Project-based Learning (PBL) is the focus of a majority of the curriculum/lesson planning work.4 Approaches to curriculum products vary with a wide range of terms being used (See sidebar on

4 Most projects are designed project-based learning approaches using the Buck Institute as a resource and provides training as a partner in several projects. The Buck Institute for Education (BIE) defines PBL as a “teaching method in which students gain knowledge and skills by working for an extended period of time to investigate and response to a complex question, problem or challenge”(BIE, 2014).



next page). However, the focus is on implementation of new mathematics and science standards, especially in light of few curricular resources or textbooks in either content area and uncertainty regarding district adoptions and rollout plans for new standards. A few partnerships are creating strategically sequenced sets of enrichment course offerings for students that will lead to the selection of university majors and careers in the STEM fields or additional STEM electives.

In the proposals, there was ambiguity in terms of how many lesson plans or modules were to be created by the end of the project in some and in others, very specific plans. This is a component of the local evaluation that continues to be refined based on the identification of models for what each partnership envisioned and samples collected as they are developed by project partners and teacher participants.

In the beginning, partnerships were at the initial stages of development of these products and the terms used began to have more definition. In March 2015, partnerships provided a sample of the curricular product that was being developed and as Cohort 10 implementation progresses, the partnerships are achieving clarity about what they are expecting teachers to produce, refine and disseminate.

The development of curriculum in the time allotted for teachers has been a challenge, but continues to be pushed as part of the professional development in both the summer and during the school year. School year activities usually focus on teaching, reflecting and refining lessons developed through either lesson study or other collaborative structures. While the development of the units can be a challenge, teachers often report the experience of teaching and trying new approaches as one of the most valuable aspects of the training. A potential result of this work, however, could be teacher-tested models for statewide dissemination and examples of implementing new standards in a STEM context.

Nearly all partnerships planned to either display or collect products in an online format and many had already begun to do so at the end of the first year of implementation. Several partnerships are exploring different ways to do this, and there are multiple Web-based technologies under consideration. This is an aspect of implementation that will continue to develop and need further tracking and review.

Rigorous and Challenging Curriculum for Students and TeachersThe mathematics partnerships reported that there is more of an interdisciplinary approach to teaching mathematics because of the professional development. Teachers are now addressing the “how” and “why” in mathematics. One project director explained that the focus is on integrated computer programming and robotics activities, while other partnerships are focusing on curriculum, hands-on and collaborative learning strategies. A project director stated that the relationship between the district and its IHE partner

Example Curriculum Products• STEM-integrated course curricula,

modules, project-based units, unit of study and integrated units of instruction.

• Units of study available in lesson bank for teachers.

• Replacement units. • Integrated engineering replacement

modules—a short-term module (1-2 weeks) and a long-term unit (3-6 weeks).

• Practice-based, project-based unit of instruction for each grade.

• Cross-curricular projects and performance tasks.

• Engineering challenges into integrated “mini” STEM units.

• STEM-integrated mathematics and science curricula and lesson modules for engineering processes.

• Integrated Math, Science and Technology Project-Based Learning (PBL) units and modules.

• Project-based, inquiry-based instruction using Mathematics Anchor Tasks (created by teachers on district-level curriculum teams) and Supporting Tasks (created by teachers at the site level).

• Mathematics lessons using Ed Puzzle. • Performance tasks/assessment.• Math/science integrated curriculum

modules piloted through lesson study. • Integrated Curricular Modules comprised

of lessons developed, tested and refined during the Lesson Study process.


Public Works

has deepened with the district seeking their assistance in integrating the Common Core with computer programming and robotics.

The teacher survey indicates that almost all participating teachers are confident in their preparation to support students in most aspects of traditional teaching such as handling discipline problems (99%), content knowledge to be creative with instruction (98%), making a difference in

their students lives (96%) and have the ability to teach all students to high achievement levels (94%). However, in other areas, there are fewer teachers that agree they are prepared to integrate curriculum across STEM disciplines (79%), integrate engineering practices (73%), and help students prepare for STEM post-secondary and career options (61%). CaMSP continues to provide unique opportunities for teachers to better understand engineering and the engineering practices incorporated within the NGSS. This survey emphasizes how new the concepts of both integrated STEM, engineering and modeling are for teachers and that the support provided through CaMSP brings these concepts to the forefront of professional development.

Partnerships that focus on both mathematics and science content reported that they are conducting cross-curricular planning. To further push the content, one partnership has created a discussion center that focuses on mathematics and science. Another project director explained that CaMSP in allowing for the inclusion of science in the school day with a strong emphasis on the use of science, engineering, mathematics practices, and technology in the classroom. Project directors report that teachers are observing more student engagement, students are more involved in the engineering challenges, and therefore teachers have increased the time spent on science instruction.

Project directors report the use of strategies such as project-based learning (PBL), technology and applying mathematics and engineering practices on a regular basis, and more teaching using inquiry-based curriculum that integrated the new mathematics and science standards has been observed throughout classrooms of participating teachers. Another project director made the observation when visiting classrooms, that teachers are strongly focused on the instructional shift through the use of PBL units and teachers are depending less on a standard lecture type format. According to one project director, teachers, over the course of a year, have deepened their mathematical understanding and strengthened their focus on English Learners.

In the Cohort 10 science partnerships, project directors have reported that students have gone on field trips that expose students to science/STEM careers. One science partnership reported that their students have participated in local robotics tournaments and another partnership described how the project has strengthened career pathway offerings in the region. One project director mentioned that students are now expected to understand the “how” and the “why” and “three-dimensional” learning that is embedded in NGSS. Another project director explained that students have become more engaged in engineering design practices, due to teachers participating in professional development training and supporting an existing module-based curriculum. One project director described the addition of a robotics course in a participating high school.

Top 3 Ways CaMSP Supports Students1. Increase student interest in

mathematics or science (89%)2. Ability to investigate STEM through

real life problems (84%)3. Ability of students to integrate STEM

across disciplines (79%)

Source: 2015 Teacher Survey (n=1,113)

I was dissatisfied with the science I was having to provide to my 6th grade students. I was only providing it, and they were not experiencing science. Now this past year, I was able to have them experience science.




Findings for Key Feature 4: Institutional Change and SustainabilityCaMSP was intended to become a better way to provide support to local school districts in connecting with college and university professors and resources for teachers and student learning. CaMSP was also intended to have spill-over effects on the Institutions of Higher Education (IHE) themselves through institutional change in the way they prepared and supported pre-service teachers and recruit and retain undergraduate students. The lead LEAs in CaMSP partnerships have direct responsibility for management and coordination of the grant as well as fiscal responsibility, which has led to more input and a better sense of ensuring that teacher needs are met in a way that secures their commitment over the three-year period. However, in terms of institutional change at the IHE, there is little evidence that partnerships are having a long-term impact on pre-service training and the like.

Despite the lack of core components of institutional change in preparing teachers for the classroom, there are strong and deep personal and collegial connections being made throughout all cohorts of partnerships that have been funded through CaMSP. Professors and instructors from colleges and universities almost universally indicate their enthusiasm for participating in the projects and what they learn from their colleagues in K-12 schools. Their participation is affecting the way professors are delivering their own content and pedagogical strategies they use to reach college students.

In terms of sustaining these kinds of relationships and support, most partnerships are relying on the actual training and transformation of participants and IHE providers, because of their involvement in the project, to change their practices and the practices of other colleagues. Some are planning teacher and student showcases to disseminate what they are doing. Use of technology is a large component of most professional development models, which will result in curriculum products. The technology strand is also most partnerships’ dissemination and sustainability component.

In the first part of implementation, dissemination has focused on Websites where teachers post and share lessons, artifacts and presentations at conferences. Websites/portals mentioned include: Haiku, Share My Lesson, My Big Campus, Google Drive, Schoology Groups, School Loop, Edmodo and Brokers of Expertise. Some partnerships will have digital archives of professional development training, including summer academy and Saturday workshop videos. Other partnerships are recording in-classroom video clips of teaching lessons. Examples: “classroom-level videos of selected K-6 STEM lessons and activities.” A few IHEs are creating courses and/or masters programs for teachers.

Impact on Course Offerings and Instruction at the IHEMany of the Institutions of Higher Education (IHE) have incorporated some of the activities from the professional development into their pre-service courses for new teachers. For example, one IHE has incorporated the disciplinary core standards for NGSS and has modeled the changes of integrated STEM teaching methodologies, and developed NGSS-based lesson plans that incorporate these dimensions and engineering to their students. Other IHEs are adding an engineering component to science courses. Project directors reported that a few IHEs have said that they will develop new teacher education courses that instruct pre-service teachers on STEM methodologies. Based on the working relationship developed with the professional development providers, several IHEs’ faculty from departments of Biology, Physics, and Chemistry have had professional development training that focuses on NGSS.

Having taught for nearly 30 years, this has gotten me to speak less, teach less, and getting the kids to work on things and solve problems together more rather than always having me just show them how to do it.



Public Works

SustainabilityAt the end of the first cycle of funding from the grant, partnerships are starting to think about what can be sustained once the grant has ended. The majority of partnerships felt that the coaching support would be useful to continue or expand throughout the district. Classroom followup activities such as lesson study or unit study could also be useful and expanded to an entire grade level or as a continuation of professional development training for teachers as well as what teachers are learning related to technology integration. The addition of STEM labs and academies, new elective courses/college courses, alignment of NGSS into science courses, and local clubs and family STEM nights could continue after the grant. Collaborative relationships with the RCP and the dissemination of the activities to a broader audience will also be sustained after the grant is finished. As the partnerships have generally had a high level of success in recruiting and retaining teachers to continue to participate and become leaders in implementation of the new mathematics and science standards, this group of teachers can provide additional support in implementation of the new standards through interactions with their colleagues both within and outside of the grant. Curriculum units and lessons will also exist to disseminate and use after the life of the grant.



Findings for Key Feature 5: Evidence-based Design and Outcomes

With the coordination of both the state and local evaluations by PW under Cohort 10, there is now a strategy in place to collect similar data across all sites as well as to create customized instrumentation and data collection for each partnership’s individual model and approach. Data from the state evaluation that is relevant at the partnership level is disaggregated and analyzed for each partnership. This includes the core and regional collaborative partner survey, the annual participating teacher survey, and state student outcome data including the SBAC for mathematics and CST for science.

In addition, under Cohort 10, PW centralized the administration of the teacher content assessment. Depending on the focus of the partnership and/or the subject areas of participating teachers, partnerships selected the Learning Mathematics for Teaching (LMT) instrument developed by the University of Michigan using its online Teaching Knowledge Assessment System (TKAS) and/or a PW-developed content assessment for science aligned to the Next Generation Science Standards, the Teacher Content Assessment for Science (TCAS).5 The next section of the report provides the first year of results from the LMT and TCAS and student outcomes for the 2015 administration of the mathematics SBAC assessment and the science California Standards Test (CST).

Mathematics partnerships were comfortable with administering the LMT as the teacher content assessment whereas science partnerships were not as comfortable with the new teacher science content assessment. Although the partnerships selected science and in most cases, engineering, many are not familiar with the Engineering Practice standards in the NGSS. The teacher content assessment that PW has developed is raising awareness of the engineering standards and how they relate to professional development.

Initially, PW visited each project to better understand the direction of the professional development and to develop a local evaluation plan as a blueprint to guide the partnerships over the next three years. In the first phase of the evaluation plan, the focus was on instrumentation needed and on initial data collection in the summer and fall of the first year of implementation.

In September 2014, evaluation plans developed initially from the partnership proposal and refined during the site visit were finalized with partnership input and submitted to CDE’s STEM Office as an attachment to the partnership’s first quarterly Year to Date (YTD) report. These plans included a variety of methods, including teacher surveys, student STEM interest surveys, student assessment embedded in curriculum products, coaching and classroom observation tools, lesson study rubrics and district student benchmark systems. PW coordinated the development of these tools and collection of data for the customized components of the local evaluation directly with each partnership and provides regular updates to CDE as part of the YTD reports that each partnership submitted.

As the state and local evaluation component of CaMSP was revised and refined under Cohort 10 to a more centralized approach managed by PW, it became evident from the site visits and review of proposals

5 Additional information about the teacher content assessment can be found at the PW website: http://www.publicworksinc.org/pw/camsp/projsupp/conassess/

Instruments & Measurement Tools in Local Evaluation Plans for Cohort 10 partnerships• Teacher surveys (100%)• Classroom observation rubrics

(85%)• Lesson study rubric (40%)• Teacher reflections (40%)• Project-specific teacher content

assessment (30%)• Student interest surveys (60%)• Project or local student

assessment (50%)

Source: Local Evaluation Plans (n=20)


Public Works

that the evaluation section of the RFA was a difficult one to write. Therefore, PW found a wide range of expectations and plans that were revised and refined in the spring 2014. Most partnerships believed they were conducting their own local evaluation, not Public Works. Most partnerships over-promised in terms of evaluation strategies. The revised plans developed as part of this process reflect the new instruments needed and the data collection plans for Years 1, 2 and 3.

Most partnerships are interested in student and teacher knowledge of and interest in STEM-related fields to be measured by survey instruments that have been customized for the partnerships. There is a great variation in terms of student outcomes. Some partnerships are not examining student work in Year 1 while others are extensively reviewing student work. For example, “a performance task or real-world application using a rubric, discuss student thinking, student misconceptions and how to provide effective feedback to students to help them improve in their reasoning, inquiry, conceptual understanding, and literacy skills.” Most have embedded student assessment as a component of the products that are under development. However, this is a continuous process of refinement and development, especially as the partnerships have more clarity on the assessment components of the curriculum products they are developing. While some partnerships have developed rubrics and tools to analyze developed lesson plans, all projects are being asked to create a rubric with PW support for analysis and refinement in Years 2 and 3. In addition, there continues to be a need for lesson study and coaching tools to quantify trends and instructional implementation.

SummaryTaken together, information gathered from partners, partnership directors, IHE partners and teachers involved in CaMSP, there are strong working relationships within the partnerships that are encouraged about what they can accomplish through the professional development support and the connections being made with each other and in the region. There is also a sense of great respect among the IHEs and the school districts and teachers with whom they are making connections.

Partnerships and participants have seen their county offices as resources in the region, which has helped garner interest in the grant and enhance the support the county office and other providers can offer. Partnerships have provided a rare opportunity for teachers in rural areas to collaborate with other teachers, especially with new standards in mathematics and science. Partnerships also reported that field experiences for teachers established relationships with professionals throughout their area. These experiences also resulted in more student engagement and opportunities to see students talk about and engage in mathematics and better understand the underlying principals of science disciplines and NGSS.

By centralizing the local evaluation component of the projects, partnerships were asked to more clearly define professional development approaches and identify measurement tools to show impact and refine implementation. Coaching models for classroom followup measured this component through observations of teaching, collection of evidence of student learning and use of formative assessments or other tools. Lesson study captured the process for developing a lesson based on student goals, measured through participant observations, revision of the lessons and feedback from teachers regarding the process. Working together as a team, partnerships viewed their work as a research project in which they are learning about and designing professional development with evidence from a model.

As implementation progressed, partnerships indicated with more clarity how they were integrating engineering and how they had benefited from the new connections they had made to engineering departments on IHE campuses. This continues to grow and develop and is a benefit of the STEM approach and easily connected by teachers to elements of NGSS standards and the Common Core, particularly the emphasis on and more formal ideas of modeling. Partnerships have also refined how they



approach the development of curriculum and other products, as well as post the material for sharing among participants and a wider audience.

Technology support for teachers varied widely in approach and emphasis, generally supporting teachers in the use of technology in teaching and collaboration and exposure to new ideas and applications of technology in the classroom. A few projects focused on computer science, robotics or app creation. Strong coaching and connections to partners during the professional development in the summer and throughout the school year provided opportunities for teachers to use the technology they were exposed to, with the benefit of coaches, facilitators and others to support them. As this cohort of partnerships emphasized the idea of an integrated STEM approach, these partnerships began with ideas about the integration of engineering and technology within a model for mathematics and science teaching—which was a very new idea within the community of professional development providers. Even for those that have been innovative in previous efforts related to mathematics and science, STEM brought in new partners, such as engineers, programmers and others who had not worked previously with K-12 teachers providing an opportunity to try new things. With the emphasis in new mathematics and science standards on both teaching practices and engineering as problem solving, these partners provided a new lens on how to deliver these ideas and make them come alive for teachers and students.

x2 • y2 = z2

F=ma

Al2(SO4

)3

79

Au197.0

F = Gm1 m

2 /r 2

A = πr2

Section 5:

Statewide Measures

& Outcomes


Public Works

Section 5: Statewide Measures & OutcomesUnder the NCLB legislation and the MSP program, CaMSP partnerships have focused on understanding baseline student achievement and teacher workforce data in order to document improvements in student achievement in mathematics and science over time. Each year, partnerships must report on two key outcome measures: (1) changes in teacher content knowledge based on a pre- and post-assessment and (2) proficiency levels on state-level assessments of students of participating teachers. These outcome measures are analyzed in this section to provide an overview of outcomes across the state and are also analyzed and reported for each partnership in the Annual Performance Report (APR).

In the transition to local evaluations coordinated by PW, the STEM Office directed partnerships to use consistent measures of teacher content for both mathematics and science for reporting on each partnership’s APR, which provided an opportunity to examine progress across the state. To align as closely as possible with the spirit and intention of CaMSP and the transition to new mathematics and science standards in California, PW selected the Learning Mathematics for Teaching or LMT measure for mathematics-focused partnerships. Because of the lack of consistent and/or appropriate measures of teacher content knowledge in science, PW, in collaboration with IHE professors and NGSS experts, developed an NGSS-aligned Teacher Content Assessment in Science (TCAS) that began with the initial implementation of Cohort 10 partnerships and has continued to be developed and used under Cohorts 11, 12 and 13. This report provides the first year analysis of pre- and post-content assessments for teachers in both mathematics and science for Cohort 10 CaMSP projects. A separate report with detailed information about the analysis of results is available for both the mathematics and science teacher content assessments.

For each year of the statewide evaluation, PW has conducted a student outcome study with the objective of determining whether the CaMSP program impacted student outcomes on state assessments in mathematics and science compared to what students would have achieved had they not been taught by teachers participating in CaMSP professional development, using a matched treatment group of teachers and their students. Up until the suspension of the California Standards Test (CST) in mathematics and English language arts in 2014, the CaMSP student outcome study had used results from the CST for both mathematics and science. In the transition to new standards under the Common Core State Standards initiative and the development of aligned assessments to be used by multiple states opting to participate in the Smarter Balanced Assessment Consortium (SBAC), CaMSP has shifted its analysis to the first year of data available for the SBAC in 2015 in mathematics in grades 3 to 8 and 11 and continues to collect and analyze the science CST, administered in grades 5, 8 and 10.

For CaMSP Cohort 10 projects, statewide analysis and reporting of both teacher content assessment and student proficiency in 2014-15 reflects each partnership’s professional development emphasis in mathematics and/or science content and the supported grade levels of participating teachers. Within Cohort 10, four partnerships selected science only as the core discipline, five partnerships selected mathematics only and the remaining 11 partnerships selected both science and mathematics as core disciplines. The analysis of teacher content assessment and student proficiency is based on the sub-set of partnerships selecting either one or both of the core disciplines for their professional development model.

Mathematics Teacher Content Assessment ResultsLearning Mathematics for Teaching (LMT) BackgroundIn the initial planning for the transition to identifying appropriate instruments for the centralized administration of the teacher content assessment for use in reporting in the APR, PW conducted a review of measurement instruments for possible use with the CaMSP local evaluation in order to make

Statewide Measures & Outcomes


a recommendation to CDE of the most reliable, valid, research-grounded and appropriate to collect common and comparable data at the local level. PW reviewed the literature and two existing compendia of information describing instruments in use at MSPs across the country, which summarized how the instruments were used and identified characteristics of the reliability, validity and the research base.1 After selecting a list of instruments for investigation, further research was conducted into each instrument to provide a fuller view of the possibilities and limitations.

The most widely used and well-known instrument in the list of teacher mathematical content knowledge assessments is the instrument developed by the Learning Mathematics for Teaching (LMT) project at the University of Michigan, which was founded in 2000 by researchers at the University of Michigan in an effort to build a questionnaire that measured pedagogical content knowledge within the domain of mathematics. Rather than incorporating items that can determine whether teachers can perform mathematical computations (as in most exams), the LMT measures a teacher’s ability to perform the tasks associated with teaching mathematics; tasks such as, the ability to evaluate student work, identify student errors, and explain numbers, operations, and mathematics procedures. For example, a test item might indicate a potential error a student has made and ask the teacher to identify the error. This is a task that mathematics teachers must reliably perform when grading homework, classwork and exams and is a good indicator of not just whether they can understand the mathematics (as in content focused exams) but also evaluates whether a teacher can identify common errors a student might make so that it can be addressed and remediated.

The intent of the conception of the LMT was to create an instrument that could measure the effects of professional development on mathematics instruction. In the 16 years since the design of this instrument, the educational researchers at the University of Michigan have heavily vetted the measure using data collected throughout the country. Item analyses were routinely conducted and the instrument was refined over time. Thus, there is good confidence that the LMT is a robust assessment with excellent reliability and validity.2

The sections of the LMT utilized for the CaMSP study include:• LMTNumberConceptsandOperations(NCOP)3 • LMTPatterns,FunctionsandAlgebra(PFA)4

LMT AnalysisAcross the ten Cohort 10 partnerships that administered the LMT, a total of 347 teachers who took both the pre- and post-assessment were analyzed. Cases that were missing either pre or post scores were deleted. This resulted in the listwise deletion of 93 cases that did not have a pre-, or in most cases, post-assessment score. This data was analyzed using four grade spans (K-2, 3-5, 6-8, 9-12), number of years teaching was sorted into five-year increments (0-5 years up to 36+ years).

The creators of the LMT have done extensive analysis on the instrument to equate the various forms of the measure. Their analyses have produced scale scores generated through the application of Item Response Theory (IRT) models. The LMT results are presented using the equated IRT scores.

1 The review of instrumentation included a Compendium of Research Instruments for STEM Education Part I: Teacher Practices, PCK and Content Knowledge and Compendium of Research Instruments for STEM Education and Part II: Measuring Students’ Content Knowledge, Reasoning Skills, and Psychological Attributes, both compiled by ABT Associates, a research firm contracted by the MSP program to provide data, research and APR reporting support.

2 The primary study conducted by the assessment creators at the University of Michigan on the reliability and validity of the instrument is documented and can be found at http://www.umich.edu/~lmtweb/files/hillshillingball.pdf. IRT reliabilities range from .71 to .84.

3 Versions MS_NCOP_2007 & EL_NCOP_2008 forms A and B, computer adaptive

4 Versions MS_PFA_2007 & EL_PFA_2006 forms A and B, computer adaptive


Public Works

To calculate changes in scores from pre- to post-assessment, the pre-IRT scores were subtracted from the post-IRT scores. Then descriptive statistics were run on the change variables. Paired samples t-tests were calculated to determine whether the differences between the means at pre- were statistically significantly different from the average scores earned at post-assessment. Finally, multiple comparisons were conducted to determine whether certain factors of interest had an effect on the outcome variables (e.g. education level, years teaching, grade level).

The following table provides information about the frequencies and descriptive statistics that were run using the variables below:

Frequencies Descriptive Statistics

• Partnership• Region• Grade Level• Gender • Ethnicity• Years Teaching• Years Teaching in District• Education Level

• Average IRT Score NCOP pre-• Average IRT Score NCOP post-• Average IRT Score PFA pre-• Average IRT Score PFA post-

Demographic Information for the LMT SampleTen projects were included in the analysis. The largest percentage of teachers were from Salinas City. San Rafael City contributed the smallest number of participants. There were a total of 347 teachers who completed both pre- and post-LMTs.

• Themajorityoftheteachersinthisanalysisweresecondaryteachers(56%).Almosthalf(44%)teach at elementary schools. The majority (72%) were female and White (58%).

• Whenaskingteachershowlongtheyhavebeenteaching,themostfrequentlyselectedresponsewas less than 5 years. The average number of years teaching was approximately 12 years and the average number of years teaching in the district was 10 years.

• NearlyhalfoftheteacherssurveyedhadaBachelor’sdegreeplus30ormoresemesterhours.And almost equal numbers had a Bachelor’s degree with no additional semester hours or a Master’s degree.

• Onaverage,teachersinthissamplehadcompleted64intensiveprofessionaldevelopmenthours, 27 followup hours, and 91 hours total in Year 1 of their CaMSP project.

LMT PerformanceAnalysis of the Number Concepts and Operations (or “Numbers”) portion of the LMT revealed that teachers improved from pre- to post-assessment (.33 to .56). A paired samples test was conducted to determine if this change was statistically significant. The paired differences were tested and it was found that there was a statistically significant difference between the pre- and post-test scores measured at the .05 level (t(346) = 6.105, p = .000).

Analysis of the Patterns, Functions, and Algebra (or “Algebra”) portion of the LMT revealed that teachers improved from pre- to post- on this portion of the exam as well (.15 to .32). A paired samples test was conducted to determine if this change was statistically significant. The paired differences were tested and a statistically significant difference between the pre- and post-test scores measured at the .05 level (t(344) = 4.257, p = .000) was found.



LMT SummaryFrequencies, descriptive statistics, paired samples tests, and multiple comparison tests were conducted to capture a profile of Cohort 10 teachers who completed the LMT, determine how they performed on the pre- and post-assessments, and whether three teacher characteristics (years teaching, educational level and grade level taught) had any impact on overall performance on the exams.

• TeachersimprovedtheirperformanceontheNumberConceptsandOperationsportionoftheLMT from pre- to post- administration to a statistically significant degree, with the same being true for the Patterns Functions and Algebra portion of the LMT.

• Multiplecomparisonanalysesrevealedthattherewerenostatisticallysignificantinteractionsbetween several variables tested, including: years teaching, education level, and grade level taught and growth on the LMT.

• Disaggregationofthechangedatabypartnershipshowedthatthereweresomedifferencesamong the partnerships in the degree to which teachers improved on the LMT. Shasta and Sacramento COE demonstrated the greatest amount of growth from pre- to post- on the assessment. This, however, could have been influenced by the fact that their scores were lower to begin with.

• Significancetestsmeasuringtheimprovementinscoresfrompre-topost-revealedstatisticallysignificant change for Escondido, Hacienda La Puente, Sacramento COE, Salinas City, Shasta COE, and marginally significant at Yolo COE. Shasta COE, Fortuna and San Joaquin secondary teachers performed at the highest levels on the post-assessment.

Science Teacher Content Assessment ResultsTCAS BackgroundThe Teacher Content Assessment for Science (TCAS) was developed by PW, in conjunction with IHE professors and NGSS experts, to provide a Next Generation Science Standards (NGSS)-based measure that could fulfill the need for a common assessment among California Mathematics and Science Partnership Grant recipients focused on science. The assessment contains items from earth, life, and physical science. It includes both multiple-choice and constructed-response items. There are different assessments based upon the following grade spans: K-2, 3-5, 6-8, and 9-12. The high school grade level assessments also contain more engineering items to align with the NGSS standards for high school. The TCAS is administered via paper and pencil. The K-8 measures consist of 25 multiple choice items and five problem-based, constructed-response items while the 9-12 grade measure consisted of six problem-based, constructed-response items. The pre-assessments were administered in the summer of 2014 and the post-assessments were administered in summer of 2015, after a full year of exposure to the Cohort 10 partnership programs. Because of the emphasis on alignment to new standards and the expertise of the development team, the assessment has validity in relation to the NGSS. The assessment is currently in the beginning stages of reliability testing.

TCAS AnalysisThe analysis began with conversion of pre- and post-TCAS Excel worksheets into SPSS so they could be merged and analyzed. Variables were re-labeled with a pre- or post- designation and then merged into the same file on Teacher ID number. Cases that were missing either pre- or post- scores were deleted.5 The resulting file contained scores for 686 teachers who took both the pre- and post-assessment across the state of California. Several categorical variables were created using scale scores of existing variables for ease of interpretation. For example, grade levels were sorted into four categories (K-2, 3-5, 6-8, 9-12). Number of years teaching was sorted into eight categories in five year increments (0-5 years up to 36+

5 This resulted in the listwise deletion of six cases that did not have a pre-assessment score.


Public Works

years). Education level and Years Teaching were also sorted into dichotomous categories. The following frequencies and descriptive statistics were run on the variables below:

Frequencies Descriptive Statistics

• Partnership• Region• Grade Level• Gender• Years Teaching• Years Teaching in District• Education Level• Ethnicity

• Number Correct, Multiple Choice, Pre-• % Correct, Multiple Choice, Pre-• Number Correct, Open Ended, Pre-• % Correct, Open Ended, Pre-• Number Correct, Total, Pre-• % Correct, Total, Pre-• Number Correct, Multiple Choice, Post-• % Correct, Multiple Choice, Post-• Number Correct, Open Ended, Post-• % Correct, Open Ended, Post-• Number Correct, Total, Post-• % Correct, Total, Post-

To calculate changes in scores from pre- to post-assessment, the pre-score percentages were subtracted from the post- score percentages. Then descriptive statistics were run on the change variables. Paired samples t-tests were calculated to determine whether the differences between the means at pre- were statistically significantly different from the average scores earned at post-assessment. Finally, multiple comparisons were conducted to determine whether certain factors of interest had an effect on the outcome variables (e.g. education level, years teaching, grade level).

Demographic Information for the TCAS Sample• Therewere15projectsincludedintheanalysis.TheCohort10partnershipsweredistributed

throughout southern, central and northern California. The number of teachers taking the assessment ranged from 23 to 70 by partnership. There was a total of 686 teachers who completed both the pre- and post-TCAS.

• Themajorityofteacherstaughtgradelevels3-5,followedby6-8,9-12,andK-2withthelowest percentage of participants. Nearly 72% of the teachers were female.

• Alittlelessthanhalfoftheteachersinthesamplehavelessthan10yearsofteachingexperience. Slightly more than half have been teaching for more than 10 years. Most teachers were in the 6 to 15 year range in regard to experience. The average was 12.7 years teaching and 10.6 years teaching in their current district.

• ThemajorityofteachersincludedinthesamplehadaBachelor’sdegreeplus30ormoresemester hours. Thirty-six percent had earned a Master’s degree or higher. Two teachers indicated that they had less than a Bachelor’s degree.

• Onaverage,theteachersinthesamplehadcompleted62hoursofIntensiveCaMSPprofessional development, 26 hours of followup, and a total of 88 hours.

TCAS PerformanceAnalysis of the Percent Correct on the multiple choice portion of the TCAS revealed that teachers did not perform as well on the post-assessment relative to the pre-assessment. The pre-assessment average score was 63% which decreased to 59% at post-assessment. This difference was statistically significant at the .05 level (t(554) = 6.56, p = .000).

In contrast, analysis of the Percent Correct on the open-ended or constructed-response portion of the TCAS revealed that teachers performed slightly better on the post-assessment relative to the pre-



assessment. The pre-assessment average score was 56%, which increased to 62% at post-assessment. This difference was statistically significant at the .05 level (t(682) = -8.087, p = .000).

Analysis of the Percent Correct on the Total on the TCAS revealed that teachers performed about the same on the pre-test relative to the post-test overall. The pre-assessment average score was 61% and 60% at post-assessment. This difference was not statistically significant.

TCAS SummaryFrequencies, descriptive statistics, paired samples tests, and multiple comparison tests were conducted to capture a profile of Cohort 10 teachers who completed the TCAS, determine how they performed on the pre- and post-assessments, and whether three teacher characteristics had any impact on overall performance on the exams.

• Ingeneral,teachersperformedbetteronthepre-assessmentrelativetothepost-assessmentonthe multiple choice portion of the exam, which was only taken by the lower grade levels.

• Teachersperformedslightlybetteronthepost-assessmentrelativetothepre-assessmentontheconstructed-response items.

• Thesedifferencesappearedtowashoutwhenanalyzingthetotalpercentageofitemscorrectasthere was no statistically significant difference between pre- and post-assessment overall.

• Teacherexperienceappearedtohaveanimpactonteacherperformance,asteacherswhohad more experience demonstrated greater gains from pre- to post- relative to those less experienced.

• Teachereducationlevel(asitwasdisaggregateddichotomously)didnotappeartohaveanimpact on teacher performance on the TCAS.

• PartnershipswiththegreatestincreasewereWestContraCosta,PasoRoblesandSanRafaelCity, but partnerships varied widely in initial scores. Shasta COE, San Rafael, Anaheim and ABC were partnerships with the highest post- scores in a relatively smaller range.

• Therewerealsodifferencesfoundwhenanalyzingtheimpactgradelevelmayhavehadonperformance. Teachers who taught the lower grade levels appeared to demonstrate greater growth over the course of the year, however this effect could have occurred because teachers who taught the upper grade levels performed at a much higher rate on the pre-assessment and may have regressed toward the mean on the post-assessment. This should be investigated further in future analyses with additional cohorts and additional years of data.

Student Outcome Study ResultsFor the student outcome study, PW designed a matched-comparison student outcome study. Participating teachers who have completed the required 84 grant hours (60 intensive hours and 24 hours of followup in Year 1) are referred to as the treatment group. After the treatment group was identified at the conclusion of Year 1 professional development activities, Public Works identified a control group of teachers matched by years of teaching, grade level taught, and educational level within the partnership participating LEAs. Proficiency levels for students of participating teachers are measured using the following two statewide assessments:

• Formathematics,studentoutcomeshavebeenmeasuredusingthemathematicsSmarterBalanced Assessment Consortium (SBAC) assessment. SBAC is a research consortium funded by the US Department of Education to develop an assessment system based on the new Common Core State Standards (CCSS) in mathematics and English language arts. Referred to as the SBAC, or CAASPP (California Assessment of Student Performance and Progress), the assessment is administered to all students enrolled in 3rd through 8th grades, and in


Public Works

11th grade. It is used to measure standards-aligned mathematics content knowledge. The test items are based on standards for each grade level in elementary and middle school, while the high school test is separated by subject-matter concept (e.g. Number and Quantity, Algebra, Geometry).

SBAC results are reported either as scaled scores or achievement levels. Scaled scores range from 2189 in 3rd grade to 2862 in 11th grade for mathematics, and are progressively higher for each grade. Achievement levels are, from lower to higher achieving: “standard not met,” “standard nearly met,” standard met,” and “standard exceeded.”6

• TheScienceCaliforniaStandardsTest(CST)isastate-levelassessmentdevelopedspecificallyto assess students’ achievement of California’s academic content standards in science that were adopted in 1998 prior to the adoption of the Next Generation Science Standards (NGSS) in 2013.7 Based on California’s No Child Left Behind (NCLB) waiver, the science CST will continue to be administered as part of its accountability system until new assessments aligned to the NGSS are developed and administered in California. According to a timeline established for the NGSS rollout anticipates that 2018-19 will be the first year of testing aligned to NGSS.

The Science CST Assessments are multiple-choice tests and administered to students in 5th, 8th and 10th grades. Topics covered vary by grade level, for example 5th covers both 4th and 5th grade standards in physical, life and earth sciences. Whereas 8th grade covers experimentation, physics concepts, and chemistry and 10th grade tests cover middle and high school level standards, as well as biology and life sciences.

Depending on grade level, all students in California complete the two assessments. For CaMSP Cohort 10 projects, statewide analysis and reporting of student proficiency in 2014-15 reflects each partnership’s professional development emphasis in mathematics and/or science content, and the supported grade levels of participating teachers. The student content assessments were analyzed for research purposes only and all individual results will be kept confidential by PW. Only pooled results are reported for the purpose of evaluating professional development initiatives, not individual teachers.

PW requested student rosters of the treatment and control teachers and combined this information with student demographic data and either SBAC results for mathematics or the California Standards Test (CST) results for science. Comparisons between students of the treatment and control teacher groups, in terms of 2014-15 SBAC scaled scores and achievement levels, were tested for significant differences. PW analyzed Science CST scaled scores and proficiency levels using tests for significant differences between treatment and control students.

Matching ProceduresSince students from the two groups (treatment and control) varied in terms of several demographic variables that are known to affect academic achievement, PW used a matching procedure called “Coarsened Exact Matching,” or CEM, to create analytic sub-samples of treatment and control students from each partnership and at each grade level. These sub-samples were considerably smaller than the entire population because they included only matched control students who were “virtual twins” of treatment students. The sub-samples were matched in terms of:8

6 For information on Smarter Balance scale score ranges, see http://www.cde.ca.gov/ta/tg/ca/sbscalerange.asp

7 The California Department of Education provides information about the NGSS rollout at the following Website: http://www.cde.ca.gov/pd/ca/sc/ngssintrod.asp.

8 Iacus, Stefano M., Gary King and Giuseppe Porro. 2008. “Matching for Causal Inference Without Balance Checking.” http://gking.harvard.edu/files/abs/cem-abs.shtml.



• Ethnicity,• Languageclassification,• PerformanceontheCaliforniaEnglishLanguageDevelopmentTest(CELDT),• Poverty,• Specialeducationdesignation,and• Giftedandtalenteddesignation.

Note that, since 2014-15 was the first year that the SBAC was available, it was not possible to match students according to their prior year mathematics performance. The CST science tests are designed to assess content knowledge in discrete areas of science and are not administered in consecutive years, thus they cannot be used to evaluate student growth. In addition, it was not possible to match students according to their prior grade level science performance.

Short of random selection and assignment to treatment and control groups, this matching method is the most robust way to account for group differences associated with achievement levels. The matched samples were used for analysis of differences between treatment and control groups.

About the Partnerships—MathematicsAcademic performance and demographic data were collected for students of both treatment and control teachers from partnerships that identified mathematics as a core discipline (a total of 16 partnerships, with 5 selecting mathematics only and 11 selecting mathematics and science), producing a database of over 44,000 mathematics students. The composition of this student population is shown in Table 5.1. As shown in Table 5.2, the smaller matched subsamples were evenly balanced in terms of all of the matching criteria.

• 559treatmentteachers(638controlteachers)• 24,487treatmentstudents(19,651controlstudents)

Table 5.1. Demographic Profile of CaMSP Mathematics Treatment and Control Students (Before Virtual Twins Matching) 2014-15

% Students (N=44,138)

Treatment (n=24,487)

Control (n=19,651)

Male 49 49

Female 51 51

Hispanic 49*** 47

African American 4 5**

White 35 35

English Only 60 62***

Limited English Proficient 18 21***

Special Education 9 10***

Gifted and Talented (GATE) 9*** 8

National School Lunch Program (NSLP) 60*** 56

*p<.05, **p≤.001, ***p≤.0001


Public Works

Table 5.2: Demographic Profile of CaMSP Mathematics Treatment and Control Students (After Virtual Twins Matching) 2014-15


Treatment (n=13,962)

Control (n=13,962)

Male 49 49

Female 51 51

Hispanic 45 45

African American 5 5

White 40 41

English Only 72 72

Limited English Proficient 21 21

Special Education 10 10

Gifted and Talented (GATE) 8 8

National School Lunch Program (NSLP) 53 53

Year 1 Student Assessment Results—MathematicsCaMSP mathematics treatment students statewide performed on average the same as the matched control students at each grade level on the SBAC, with the exception of 8th grade, in which control students did significantly better by a 12-point difference (Table 5.3). The proportions of students that met or exceeded the Common Core State Standards for mathematics were nearly identical for treatment and control groups in the elementary school grade levels. In the middle and high school grade levels, grades 8 and 11, however, there were more control students who met or exceeded the CCSS-M achievement levels than treatment students.

Table 5.3: SBAC Mathematics, Matched Treatment to Control Students, Scaled Scores and Percent Met or Exceeded Standard, Grades 3-8, & 11 2014-15

Grade Level

Scaled Score % At or Exceeded Standards

n Treatment Control Difference Treatment Control Difference

3rd 2,069 2,413 2,412 1 39% 38% 1%

4th 2,369 2,443 2,447* -4 29% 31% -2%

5th 2,454 2,471 2,475 -4 27% 29% -2%

6th 1,432 2,492 2,497 -5 29% 31% -1%

7th 1,818 2,520 2,524 -4 37% 37% -1%

8th 2,001 2,528 2,540** -12 32% 38%*** -6%

11th 1,819 2,561 2,566 -6 28% 32%** -4%

*p<.05, **p≤.001, ***p≤.0001



Figure 5.1: SBAC Mathematics, Matched Treatment to Control, Achievement Levels Grades 3-5, 2014-15

33% 34% 34% 33% 45% 44%

29% 29% 37% 36%

28% 27%

26% 27%

21% 20% 14% 15%

13% 11% 8% 10% 13% 14%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Trea

tmen

t

Cont

rol

Trea

tmen

t

Cont

rol

Trea

tmen

t

Cont

rol

3rd Grade 4th Grade 5th Grade

Exceeded

Met

Nearly Met

Not Met

Figure 5.2: SBAC Mathematics, Matched Treatment to Control, Achievement Levels Grades 6-8 & 11, 2014-15

39% 37% 35% 34% 41% 38%

43% 41%

31% 32% 28% 28%

28% 24%

28% 27%

17% 17% 21% 19%

16% 19%

18% 22%

12% 14% 15% 19% 16% 19% 10% 10%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Trea

tmen

t

Cont

rol

Trea

tmen

t

Cont

rol

Trea

tmen

t

Cont

rol

Trea

tmen

t

Cont

rol

6th Grade 7th Grade 8th Grade 11th Grade

Exceeded

Met

Nearly Met

Not Met

6th Grade 7th Grade 8th Grade 11th Grade

3rd Grade 4th Grade 5th Grade


Public Works

In addition to the overall analysis of SBAC outcome data by grade level, results were disaggregated and analyzed at the partnership level. These results were reviewed based on discipline selected for any partnerships selecting mathematics or science only compared to both. However, no discernable patterns emerged regarding core discipline selection within STEM. For partnerships at each grade level, results were mixed in terms of performance of the treatment students compared to the control group. However, a few partnerships emerged with statistically significant positive differences for the treatment group. For example, 3rd graders in Lakeside, 4th graders in Tuolumne, 6th graders in Butte, 7th graders in Fortuna and Sacramento COE, 8th graders in Fortuna and Yolo COE 11th graders. There were a few others with positive results. However, they were not statistically significant. Note that SBAC results from the 2015 administration represent both the first year of outcome data for Cohort 10 partnerships and for the state as a whole, after the field test in 2014. Much is to be learned from this assessment in California. The results represent both a baseline measure but also reflect the first year of implementation. Going forward, 2015 outcomes can be compared to 2016 outcomes using a comparable measure for mathematics.

About the Partnerships—ScienceFor science, PW re-matched treatment students to control students, due to grade level differences between the SBAC and CST assessments and to increase sample size. As with the mathematics match, academic performance and demographic data were collected for students of both treatment and control teachers, producing a database of nearly 33,000 students before matching. Partnerships included in the analysis selected either science only (4) or mathematics and science as core disciplines (11). The composition of this student population is shown in Table 5.4. As shown in Table 5.5, the smaller matched subsamples were evenly balanced in terms of all of the matching criteria.

• 338treatmentteachers(323controlteachers)• 18,044treatmentstudents(14,853controlstudents)

Table 5.4: Demographic Profile of CaMSP Science Treatment and Control Students (Before Virtual Twins Matching) 2014-15


Treatment(n=18,044)

Control (n=14,853)

Male 50 51

Female 50 49

Hispanic 60 61**

African American 5* 4.6

White 20*** 19

English Only 49*** 44

Limited English Proficient 17 18**

Special Education 8 9**

Gifted and Talented (GATE) 12** 11

National School Lunch Program (NSLP) 68* 69

*p<.05, **p≤.001, ***p≤.0001



Table 5.5: Demographic Profile of CaMSP Science Treatment and Control Students (After Virtual Twins Matching) 2014-15


Treatment (n=9,667)

Control (n=9,667)

Male 51 51

Female 49 49

Hispanic 53 53

African American 6 6

White 27 27

English Only 64 64

Limited English Proficient 22 22

Special Education 10 10

Gifted and Talented (GATE) 10 10

National School Lunch Program (NSLP) 61 61

Year 1 Student Assessment Results—ScienceTreatment students performed about the same as the matched control students in terms of proficiency levels for 5th and 10th grade science CSTs. However, control students scored on average six points higher than treatment students in 8th grade. This difference was statistically significant (Table 5.6). Although nearly identical proportions of students reached proficient and advanced levels in both 5th and 10th grade levels, more control students reached an advanced level than treatment students in 8th grade (Table 5.6). When results for the science CST are disaggregated at the partnership level, 8th grade treatment students in Fortuna and Shasta COE outperformed their control student counterparts. Similarly, 10th graders from San Joaquin COE in the treatment group outperformed the control students.

Table 5.6: CST Science, CaMSP Matched Treatment to Control Students, Scaled Scores and Percent Proficient, Grades 5 & 8, 2014-15

Grade Levels

Scaled Score % Proficient or Advanced

n Treatment Control Difference Treatment Control Difference

5th 1,954 350 352 -2 50% 50% 0%

8th 3,875 369 372* -4 55% 58%* -3%

10th 3,838 349 351 -2 48% 49% -1%

*p<.05, **p≤.001, ***p≤.0001


Public Works

Figure 5.3: CST Science, CaMSP Matched Treatment to Control, Proficiency Levels Grades 5, 8 & 10, 2014-15

8% 8% 12% 11% 7% 9%

11% 11% 13%

11% 12% 12%

31% 31% 19% 20%

33% 30%

34% 34%

22% 24%

28% 26%

15% 16%

34% 34%

20% 22%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Trea

tmen

t

Cont

rol

Trea

tmen

t

Cont

rol

Trea

tmen

t

Cont

rol

5th Grade 8th Grade 10th Grade

Advanced

Proficient

Basic

Below Basic

Far Below Basic

Summary and Interpretation of Mathematics and Science Student Outcomes The mathematics baseline between-groups comparison showed no significant difference in scale score or achievement level performance for grades three, five, six, seven, and 11. This lack of variability between treatment and control groups indicates that the matching procedure produced groups that were comparable in terms of demographics and achievement. However, despite the fact that treatment and control groups were matched on a variety of demographic variables known to influence academic achievement, the control group did outperform the treatment group by four points in 4th grade and 12 points in 8th grade. This statistically significant variability shows that, at least for 4th and 8th grade, unobserved factors must have influenced performance.

It is important to recognize, however, that the purpose of the current evaluation is to provide baseline information about the academic achievement of the students in the partnership samples. Going forward, future evaluations will take past academic achievement (prior years’ test scores) into account, so that the treatment and control groups can be matched in such a way that academic growth attributable to CaMSP professional development can be assessed.

Similarly, the science analysis showed that the control group outperformed the treatment group in 8th grade, but not in 5th or 10th grade science. The difference in 8th grade achievement is small (3%). Since CST science tests always assess discrete knowledge areas, they can never be used to evaluate student growth in science and are not yet aligned to the new standards for science. Next year’s evaluation will build on the current analysis by contributing an additional year of post-program performance analysis.

x2 • y2 = z2

F=ma

Al2(SO4

)3

79

Au197.0

F = Gm1 m

2 /r 2

A = πr2

Section 6:

Conclusion & Next Steps


Public Works

Conclusion & Next StepsUnder the MSP program of the US Department of Education and through CaMSP, teachers have participated in professional development activities since 2003 that focused on evidence-based science and mathematics teaching methods to improve teacher content knowledge and student academic achievement. CaMSP participants are asked to demonstrate a sustained commitment to 84 hours of professional development each year for the three-year duration of the grant.

During the summer and throughout the school year, participating teachers are exposed to intensive professional learning and activities such as lesson study, coaching and professional learning communities. CaMSP programming is delivered by a customized partnership of university and community college instructors, professional development providers, teacher leaders and community partners. This design for professional development provides a structure to support teachers as they try to embed what they have learned about mathematics, science and STEM content during intensive professional development into the classroom.

CaMSP teachers are provided with opportunities throughout their participation to try out new student engagement strategies, observe and document student learning in new ways, practice new approaches to familiar lesson topics and provide and receive feedback in a collegial setting.

From 2003 to 2013, the California Department of Education (CDE) funded 11 separate cohorts consisting of 135 CaMSP partnerships that encompassed multiple districts, institutions of higher education (IHE), county offices of education (COE) and other professional development partners across the state. These cohorts were limited to grades three to eight for science and three to Algebra I for mathematics.

In 2014, the STEM Office funded 20 partnerships under Cohort 10 with support for more than 1,200 teachers that represented a substantial shift in focus where partnerships were directed to design professional development models for grades K-12 that integrated the disciplines of science, technology, engineering, and mathematics through a variety of models for and approaches to professional development. Cohort 10 partnerships were also asked to design curriculum products that incorporated new mathematics and science standards in order to support implementation of STEM learning within existing school structures.



Cohort 10 STEM Definition:An approach to teaching and learning that emphasizes integral connectedness of at least one of the core disciplines—science and mathematics—and at least one of the supporting disciplines—technology and engineering. The connections are made explicit through the collaboration of both educators and their students, resulting in real and appropriate contexts that are built into the instruction, curriculum and assessment. The common element of problem solving as defined in the CCSS-M and the NGSS is emphasized across the identified STEM disciplines and grade spans allowing students to engage, explore, expand and evaluate their learning and apply critical thinking skills as they learn.

STEM-ED objectives for teachers: • CreateSTEMStructure• Determinegaps• EngageinappropriatePD• Learnnewteachingstrategies• LearnandexperienceSTEMcollegeand

careers• Integrate• Create• Pilot• ImplementnewstandardsinaSTEMcontext• Becometeacherleaders(byYear3)• Facilitateimprovedstudentperformancein

STEM-ED disciplines

STEM-ED objectives for students: • Learnacrossdisciplines• Integrateacrossdisciplines• ExperienceSTEM• Engage• Inquire• Improveandsucceed• DemonstratepositiveattitudestowardSTEM

CaMSP Support for STEM Learning and Transition to New PoliciesThe transition to a STEM-focused approach to professional learning for teachers under CaMSP first implemented under Cohort 10 reflects at least two important understandings in improving schools. First, policy makers, advocates for educational equity and closing achievement gaps, and the business community continued to connect to and push publicly for improvements in the systems that prepare young people to become mathematicians, scientists, computer scientists and engineers. Second, the transition to STEM-focused professional learning for teachers combined what has been learned about high-quality professional development with a sense that all students would benefit from a more integrated approach to learning in order to have the skills and knowledge base they would need to contribute to the robust and ongoing changes in technology and a complex economy.

For California, this renewed attention to STEM occurred amidst vast changes to the accountability-based system of standards and assessments that had developed under the No Child Left Behind Act (NCLB), the 2002 reauthorization of the Elementary and Secondary Education Act. These changes included new standards and assessments for mathematics and English language arts under the Common Core State Standards initiative and subsequent adoption by California. In addition, the development of Next Generation Science Standards (NGSS) and adoption by California represented a new sense of consensus nationally about how to teach science and what is essential for students to learn throughout their K-12 experience.

While adoption of standards by California in mathematics and science established that there was a consensus at the state level about the eventual direction for local districts and schools, the actual implementation of the standards was both daunting and ambiguous for teachers and other stakeholders at the local level. Most of these efforts occurred at a time when the No Child Left Behind Act (NCLB) was characterized by policymakers, administrators and teachers as having run its course and long overdue for reauthorization. However, it continued to be the law under which state and local entities had to comply.


Public Works

As state-by-state negotiations continued with the US Department of Education regarding how to comply with the accountability provisions of NCLB and new initiatives by the Administration, Congress debated various versions of potential reauthorizations of the Elementary and Secondary Education Act (ESEA). Despite the excitement of new standards in mathematics and science that represented a substantial shift in emphasis to deeper understanding of the content coupled with more specific direction on important pedagogical considerations, teachers had few materials, and textbooks that were either not aligned to new standards or only nominally aligned.

At this time, California launched a variety of initiatives including timelines for textbook adoption and development of new curriculum frameworks in these content areas. In addition, California enacted the Local Control Funding Formula (LCFF), implemented new tests under the Smarter Balanced Assessment Consortium (SBAC) in mathematics and English Language arts and established a new accountability system under the California Assessment of Student Progress and Performance (CAASPP).

During this time, CaMSP funding and the in-depth professional development support provided one important way for school districts, particularly small and medium size districts, to partner with each other, university partners and professional development providers to make sense of the changes on the educational policy landscape. With the recent signing into law of ESEA under the Every Student Succeeds Act (ESSA) on December 10, 2015, California is devising a transition plan and anticipating further changes to accountability systems at the federal and state levels.

Opportunities from CaMSP to Support Quality Professional LearningAs the transitions at both the state and federal levels continues to evolve and impact schools, teachers and students, findings from the data collected for the state evaluation of CaMSP Cohort 10 STEM projects provide an important lens about the coming challenges and opportunities to support teachers, and, in turn, students to learn in new ways.

CaMSP combines locally customized professional development models based on research and recognized strategies to support teacher learning and classroom implementation with a longer-term horizon to improve and reflect on what is working. Partners within CaMSP have taken the opportunity to develop approaches to teacher learning that occur throughout the school year that are customized to the needs and challenges faced in partner districts. While the models and approaches are customized and adapted to local constraints and the participants that make up the cohort of teachers, they often reflect years of individual partner experience providing and improving how professional development is delivered.

The qualitative data collected for the evaluation demonstrates how this has become an organic experience that almost always reflects the consensus of researchers on what high-quality professional development should look like. For participating teachers, however, it is a novel experience that takes time to understand and implement—thus, the three year grant period provides opportunities for both providers to hone and improve on their initial models, and for teachers to provide realistic input about the expectations for implementing various ideas in the classroom. Communication and discourse, effective questioning and strategies and project-based learning are emphasized by most, if not all, CaMSP Cohort 10 projects.

CaMSP has provided opportunities for understanding engineering and integrated STEM learning using discipline-specific approaches, university expertise and community partners. The integration of engineering within CaMSP was a new aspect included in the transition to STEM. However, the timing provided an important opportunity to introduce teachers at all grade levels to NGSS engineering practices, which form a key construct for instruction and orientation to the new standards. The engineering practices are often referred to as the instructional building blocks of the NGSS in tandem



with the Standards for Mathematical Practices included in the Common Core to develop student thinking.

By requiring that university partners be involved representing all disciplines selected under STEM in designing and delivering professional development, CaMSP partnerships brought in many new professors of engineering, and, to a lesser extent, technology, to help them design and implement new activities and to develop a basic understanding of engineering and what engineers do. While this required new ways of thinking about professional development approaches, over time the integration of engineering proved to be a valuable component in teachers’ beginning understanding of NGSS in a real and more practical way. In addition, the integration of engineering and technology with regional partners and others allowed for outreach to partners for resources and assistance in planning lessons, classroom activities and, for some, job shadowing or other field experiences for teachers.

CaMSP provides real and sustained opportunities for collaboration and teacher leadership to develop and adjust professional development approaches over time to meet teacher needs. While there is a structure to the intensive professional development delivered during the summer and school year, the collaborative structures (grade level teams, lesson study groups and individual coaching support) are often the most fluid aspects of CaMSP that allow the professional development to feel customized to participating teacher needs. Most partners build these components based on previous experience, either in CaMSP or other professional development programs. More than half of the partnerships are using the lesson study approach, which has been adapted for STEM and integrated learning beyond the traditional approach of developing and refining a single lesson.

Partnerships are also creating new single-disciplinary, interdisciplinary and multidisciplinary curriculum for students. Project-based Learning (PBL) is the focus of a majority of the curriculum product work. With a focus on implementation of new mathematics and science standards, this work has been helpful to teachers, especially in light of few curricular resources or textbooks in either content area and uncertainty regarding district adoptions and rollout plans for new standards. However, it can also be somewhat overwhelming in the context of implementation of other initiatives—despite these difficulties, having the CaMSP support is consistently mentioned as beneficial and enriching by participating teachers in surveys and focus groups.

Embedding Formative and Summative Evaluation Support and Technical Assistance provides another lens to improve and fine tune implementation. In the transition to local evaluations coordinated by PW, the state’s approach to measurement and management of the CaMSP program expanded beyond consistent measures for student outcomes to a statewide approach, to measurement of teacher content knowledge. Beginning with Cohort 10, partnerships began to use consistent measures of teacher content for both mathematics and science for reporting on each partnership’s APR, providing an opportunity to examine progress across the state. To align as closely as possible with the spirit and intention of CaMSP and the transition to new mathematics and science standards in California, PW selected the Learning Mathematics for Teaching or LMT measure for mathematics-focused partnerships. Because of the lack of consistent and/or appropriate measures of teacher content knowledge in science, in collaboration with IHE professors and NGSS experts, PW developed an NGSS-aligned Teacher Content Assessment in Science (TCAS) that incorporated engineering practices.

By centralizing the local evaluation component of the projects and including both statewide measures and customized instruments designed to support each professional development model, partnerships were asked to more clearly define professional development approaches and identify measurement tools to show impact and refine implementation. Coaching models for classroom followup measured this component through observations of teaching, collection of evidence of student learning and use of formative assessments or other tools. Lesson study captured the process for developing a lesson based


Public Works

on student goals, measured through participant observations, revision of the lessons and feedback from teachers regarding the process. Working together as a team, partnerships viewed their work as a research project in which they are learning about and designing professional development with evidence from a model.

Strong Partnerships and a structure for implementation offer better chances for long-term success and retention of teachers. Taken together, information gathered from partners, partnership directors, IHE partners and teachers involved in CaMSP, there are strong working relationships within the partnerships and paarticipants are encouraged about what they can accomplish through the professional development support and connections being made with each other and in the region. There is also a sense of great respect among the IHEs and the school districts and teachers with whom they are making connections. Despite the initial learning curve for start up of a CaMSP project, retention of teachers continues to be a strength of implementation with attrition of very few teachers, usually because of changes out of the project’s control.

Partnerships and participants have seen their county offices as resources in the region, which has helped garner interest in the grant and enhance the support the county office and other providers can offer. Partnerships have provided a rare opportunity for teachers in rural areas to collaborate with other teachers, especially with new standards in mathematics and science. Partnerships also reported that field experiences for teachers established relationships with professionals throughout their area, resulted in more student engagement and opportunities to see students talk about and engage in mathematics, and better understand the underlying principals of science disciplines and NGSS.

Next Steps in the PW EvaluationIn 2015, the STEM Office issued two subsequent RFAs to fund additional partnerships. Cohort 11 began in early 2015 and included 12 partnerships modeled largely after the structure of Cohort 10. In May 2015, 12 additional partnerships were funded under Cohort 12, which retained more of an emphasis on discipline-specific professional development to support implementation of new mathematics and science standards adopted by California. Data collected for Cohort 10 and these two new cohorts will be included in the next statewide evaluation report, which incorporates implementation of CaMSP in the 2015-16 school year. PW will continue to support local partnerships in the completion of federal and state reporting requirements through the CaMSP database, state evaluation surveys, student and teacher content assessment data and customized local evaluation plans. PW communicates regularly with partnerships regarding evaluation efforts, through newsletters and the CaMSP page on the central Web site: www.publicworksinc.org.

x2 • y2 = z2

F=ma

Al2(SO4

)3

79

Au197.0

F = Gm1 m

2 /r 2

A = πr2

Appendix A:

Cohort 10Map

CaMSP 2015 Report Page A-1

Public Works

42

3

1

5

6

7

9

10

8

11

12

13

14

1516

17

1819

20

Cohort 10 PartnershipsMathematics7 Hacienda La Puente USD (HLPUSD CaMSP Project): Technology9 Lamont ESD (South Kern Math Partnership): Technology13 Sacramento COE (EMITS): Engineering20 Yolo COE (C-STEM+): Technology & Engineering

Science1 ABC USD (ABC STEM Professional Learning Initiative): Engineering2 Anaheim USD (Chapman-Anaheim Science Partnership): Technology & Engineering4 Coachella Valley USD (Project Prototype):Engineering12 Rialto USD (R-iSMART): Technology

Mathematics & Science3 Butte COE (iSTEM): Technology & Engineering5 Escondido Union ESD (Escondido STEM Iniative): Technology 6 Fortuna ESD (HISI): Technology & Engineering8 Lakeside USD (IDEAS 2.0): Technology 10 Orange CDE (SYSTEMS): Technology & Engineering11 Paso Robles Joint USD (CCSP:TLC Collaborative): Technology & Engineering14 Salinas City ESD (Salinas City Elementary Mathematics and Technology Partnership): Technology 15 San Joaquin COE (SIMMS): Technology 16 San Rafael City ESD (ITEAMS): Technology & Engineering17 Shasta COE (NSSP): Technology 18 Tuolumne COE (STEM-TRACKS): Technology & Engineering19 West Contra Costa USD (STEM West): Technology & Engineering

Key

Mathematics & ScienceMathematics OnlyScience Only

Page A-2 Appendix A: Cohort 10 Map


x2 • y2 = z2

F=ma

Al2(SO4

)3

79

Au197.0

F = Gm1 m

2 /r 2

A = πr2

Appendix B:

Bibliography

CaMSP 2015 Report Page B-1

Public Works

BibliographyACT, Inc. (2013). ACT National Curriculum Survey: Mathematics.

Aguilar, E. (2011, February 15). Coaching teachers: What you need to know. EdWeek - Teacher. Retrieved from http://www.edweek.org/tm/articles/2011/02/15/tln_coaching.html?qs=coaching+teachers+what+you+need+o+know

Albo-Canals, J., Heerink, M., Diaz, M., Padillo, V., Maristany, M., Barco, A., … Dufresne, S. (2013). Comparing two LEGO Robotics-based interventions for social skills training with children with ASD. In RO-MAN, 2013 IEEE (pp. 638–643). Gyeongju: IEEE. doi:10.1109/ROMAN.2013.6628420

Ames, C. (1992). Classrooms: Goals, structures, and student motivation. Journal of Educational Psychology, 84(3), 261.

Ames, C., & Archer, J. (1988). Achievement goals in the classroom: Students’ learning strategies and motivation processes. Journal of Educational Psychology, 80(3), 260.

Atkinson, R. D., & Mayo, M. J. (2010). Refueling the US innovation economy: Fresh approaches to science, technology, engineering and mathematics (STEM) education. The Information Technology & Innovation Foundation, Forthcoming. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1722822

Ball, D. L., Thames, M. H., & Phelps, G. (2008). Content knowledge for teaching what makes it special? Journal of Teacher Education, 59(5), 389–407.

Berry, B. (2010). Getting “real” about teaching effectiveness and teacher retention. Journal of Curriculum and Instruction, 4(1). doi:10.3776/joci.2010.v4n1p1-15

Berry, B., & Hirsch, E. (2005). Recruiting and retaining teachers for hard-to-staff schools. Washington, DC: National Governors’ Association Center for Best Practices. Retrieved from http://beta.nga.org/files/live/sites/NGA/files/pdf/0510RECRUITINGTEACHERS.PDF

Birman, B. F., Desimone, L., Porter, A. C., & Garet, M. S. (2000). Designing professional development that works. Educational Leadership, 57(8), 28–33.

Black, P., Harrison, C., Lee, C., Marshall, B., & William, D. (2002). Working inside the black box: Assessment for learning in the classroom. Granada Learning.

Bonwell, C. C., & Eison, J. A. (1991). Active learning: Creating excitement in the classroom. School of Education and Human Development, George Washington University, Washington, DC. Retrieved from http://www.ydae.purdue.edu/lct/hbcu/documents/Active_Learning_Creating_Excitement_in_the_Classroom.pdf

Borko, H. (2004). Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33(8), 3–15. doi:10.3102/0013189X033008003

Brand, B. (2009). High school career academies: A 40-year proven model for improving college and career readiness. Washington, DC: American Youth Policy Forum. Retrieved from http://eric.ed.gov/?id=ED526425

Brophy, J. (1981). Teacher praise: A functional analysis. Review of Educational Research, 51(1), 5–32.

Buck Institute for Education (BIE). (2014). What is Project Based Learning (PBL)? Retrieved from http://bie.org/about/what_pbl

Burns, A., & de Silva Joyce, H. (2005). Teachers’ voices 8: Explicitly supporting reading and writing in the classroom. Sydney, N.S.W.: National Centre for English Language Teaching and Research, Macquarie University. Retrieved from http://www.ameprc.mq.edu.au/docs/research_reports/teachers_voices/Teachers_voices_8.pdf

Page B-2 Appendix B: Bibliography


Bushweller, K. (2014, March 13). Digital advances Common Core fuel new testing approaches. Education Week. Retrieved from http://www.edweek.org/ew/articles/2014/03/13/25execsum.h33.html?tkn=NMMFrf7H0rID5yQ6qvcMBwtgRuC9NrzhpcUM&print=1

California Department of Education. (2013). Quality Professional Learning Standards. Professional Learning Support Division, California Department of Education. Retrieved from http://www.cde.ca.gov/pd/ps/qpls.asp

California STEM Learning Network (CSLNet). (2014). Regional Networks. Retrieved from http://www.cslnet.org/who-we-are/regional-networks/

Camins, A. (2001, Fall). Dimensions of inquiry. FOSS Nesletter. Retrieved January 12, 2014, from http://lhsfoss.org/newsletters/archive/FOSS18.InquiryDimensions.html

Carnegie Mellon Robotics Academy. (2013). ROBOTC VEX Cortex video trainer: Product overview. Carnegie Mellon Robotics Academy. Retrieved from http://www.education.rec.ri.cmu.edu/products/teaching_robotc_cortex/index/product_overview.pdf

Chang, K. (2011, July 11). American girls sweep Google’s first science fair. The New York Times. Retrieved from http://www.nytimes.com/2011/07/19/science/19google.html?_r=0

Chazan, D., & Ball, D. L. (1995). Beyond exhortations not to tell: The teacher’s role in discussion-intensive mathematics classes. National Center for Research on Teacher Learning, Michigan State University. Retrieved from http://education.msu.edu/NCRTL/PDFs/NCRTL/CraftPapers/cp952.pdf

Cheng, C. C., Huang, P. L., & Huang, H. K. (2013). Cooperative learning in Lego robotics projects: Exploring the impacts of group formation on interaction and achievement. Journal of Networks, 8(7), 1529–1535. doi:10.4304/jnw.8.7.1529-1535

Christensen Institute. (2012). Blended learning model definitions. Retrieved March 4, 2014, from http://www.christenseninstitute.org/blended-learning-model-definitions/

Cochran, K., King, R. A., & DeRuiter, J. A. (1991). Pedagogical content knowledge: A tentative model for teacher preparation. Presented at the Annual Meeting of the American Educational Research Association, Chicago, IL: National Center for Research on Teacher Learning. Retrieved from http://eric.ed.gov/?id=ED340683

College & Career Academy Support Network (CCASN). (2010). Career Academies: An Overview. CCASN, UC Berkeley. Retrieved from http://casn.berkeley.edu/resource_files/CareerAcademies_overview.pdf

Corcoran, T. B. (1995). Helping teachers teach well: Transforming professional development. New Brunswick, NJ: Consortium for Policy Research in Education.

Cradler, J., McNabb, M., Freeman, M., & Burchett, R. (2002). How does technology influence student learning? Learning and Leading with Technology, 29(8), 46–49.

Custer, R. L. (2009). Professional development for teachers of engineering: Research and related articles. The Bridge: National Academy of Engineering, 39(3), 18–25.

Darling-Hammond, L., & Richardson, N. (2009). Research review/teacher learning: What matters. Educational Leadership, 66(5), 46–53.

Darling-Hammond, L., Wei, R. C., Andree, A., Richardson, N., & Orphanos, S. (2009). Professional learning in the learning profession. Washington, DC: National Staff Development Council. Retrieved from http://www.learningforward.org/docs/pdf/nsdcstudy2009.pdf

Desimone, L. M. (2009). Improving impact studies of teachers’ professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181–199.


Public Works

Desimone, L. M., Porter, A. C., Garet, M. S., Yoon, K. S., & Birman, B. F. (2002). Effects of professional development on teachers’ instruction: Results from a three-year longitudinal study. Educational Evaluation and Policy Analysis, 24(2), 81–112.

Dewey, J. (1938). Experience and education. New York, NY: Macmillian Publishing Company.

Di Stefano, G., Gino, F., Pisano, G., & Staats, B. (2014). Learning by thinking: How reflection aids performance. Harvard Business School NOM Unit Working Paper, (14-093), 14–093.

DuFour, R. (2004). What is a “professional learning community”? Educational Leadership, 61(8), 6–11.

Dweck, C. S. (1986). Motivational processes affecting learning. American Psychologist, 41(10), 1040.

E-ducation. (2013, June 29). A long-overdue technological revolution is at last under way. The Economist. Retrieved from http://www.economist.com/news/leaders/21580142-long-overdue-technological-revolution-last-under-way-e-ducation

Engineering is Elementary (EiE). (2012). Research and evaluation results for the Engineering is Elementary Project: An executive summary of the first eight years. Boston, MA: Museum of Science. Retrieved from http://www.eie.org/sites/default/files/eie_executive_summary_of_research_5_29_2012.pdf

Engineering is Elementary (EiE). (2014a). FAQ’s. Retrieved from http://www.eie.org/engineering-elementary/engineering-elementary-frequently-asked-questions

Engineering is Elementary (EiE). (2014b). Our Products. Retrieved from http://www.eie.org/engineering-elementary/our-products

Engineering is Elementary (EiE). (2014c). The EIE Curriculum. Retrieved from http://www.eie.org/eie-curriculum

Feiman-Nemser, S. (2001). Helping novices learn to teach lessons from an exemplary support teacher. Journal of Teacher Education, 52(1), 17–30.

Feiman-Nemser, S. (2003). What teachers need to learn. Educational Leadership, 60(8), 25–29.

Flynn, K., & Hill, J. D. (2006). Classroom Instruction That Works with English Language Learners Book. Association for Supervision and Curriculum Development. 1703 North Beauregard Street, Alexandria, VA 22311-1714.

Futernick, K. (2007). A possible dream: Retaining California’s teachers so all students learn. Sacramento, CA: California State University Center for Teacher Quality. Retrieved from http://www.calstate.edu/teacherquality/documents/possible_dream.pdf

Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915–945.

Grabinger, R. S., & Dunlap, J. C. (1995). Rich environments for active learning: A definition. Research in Learning Technology, 3(2). Retrieved from http://www.researchinlearningtechnology.net/index.php/rlt/article/view/9606

Graham, E. (2013, April 25). “A Nation at Risk” turns 30: Where did it take us? NEA Today. Retrieved from http://neatoday.org/2013/04/25/a-nation-at-risk-turns-30-where-did-it-take-us-2/

Guskey, T. R. (2002). Professional development and teacher change. Teachers and Teaching: Theory and Practice, 8(3), 381–391.

Harrison, C., & Killion, J. (2007). Ten roles for teacher leaders. Educational Leadership, 65(1), 74.

Hart, J. E., & Lee, O. (2003). Teacher professional development to improve the science and literacy achievement of English language learners. Bilingual Research Journal, 27(3), 475–501.



Hord, S. M. (1997). Professional learning communities: Communities of continuous inquiry and improvement. Leadership, 40(1), 58–59.

Intelitek. (2009). Robotics Engineering Curriculum (REC). Retrieved from http://www.intelitekdownloads.com/REC/

Johnson, D. W., & Johnson, R. T. (1999). Making cooperative learning work. Theory into Practice, 38(2), 67–73.

Kemple, J. J. (1997). Career academies: Communities of support for students and teachers: further evidence from a 10-site evaluation. (Text). New York, N.Y.: Manpower Demonstration Research Corp. Retrieved from http://www.mdrc.org/publication/career-academies-emerging-findings

Koehler, M., & Mishra, P. (2009). What is technological pedagogical content knowledge (TPACK)? Contemporary Issues in Technology and Teacher Education, 9(1), 60–70.

Kolb, D. A. (1984). The process of experiential learning. In Experiential learning: Experience as the source of learning and development. (Chapter 2, pp. 20–38). Englewood Cliffs, NJ: Prentice-Hall, Inc.

Krishnamurthi, A., Ballard, M., & Noam, G. G. (2014). Examining the impact of afterschool STEM programs. Afterschool Alliance. Retrieved from http://www.afterschoolalliance.org/ExaminingtheImpactofAfterschoolSTEMPrograms.pdf

Kuenzi, J. J. (2008). Science, technology, engineering, and mathematics (STEM) education: Background, federal policy, and legislative action (Paper 35). Congressional Research Service Reports. Retrieved from http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1034&context=crsdocs

Kuhn, L., & Reiser, B. (2005). Students constructing and defending evidence-based scientific explanations. In Annual Meeting of the National Association for Research in Science Teaching, Dallas, TX. Retrieved from http://www.umich.edu/~hiceweb/iqwst/Papers/KuhnReiserNarst2005.pdf

Lansiquot, R. D., Blake, R. A., Liou-Mark, J., & Dreyfuss, E. (2011). Interdisciplinary problem-solving to advance STEM success for all students. Peer Review, 13(3), 19–22.

Lee, O., Quinn, H., & Valdes, G. (2013). Science and language for English language learners in relation to Next Generation Science Standards and with implications for Common Core State Standards for English language arts and mathematics. Educational Researcher, 42(4), 223–233. doi:10.3102/0013189X13480524

LEGO Education. (2014). LEGO Education. Retrieved from https://education.lego.com/en-us/lesi/

Lewis, C. (2003). The essential elements of lesson study. Northwest Eisenhower Regional Consortium for Mathematics and Science at the NWEL Mathematics an Science Education Center, 4(3). Retrieved from http://educationnorthwest.org/resource/741

Linked Learning Alliance. (2011). Pilot program. Retrieved from http://linkedlearning.org/about/where-linked-learning-is-happening/pilot-program/

Linked Learning Alliance. (2014a). About Linked Learning. Retrieved from http://linkedlearning.org/about/

Linked Learning Alliance. (2014b). Evidence of effectiveness. Retrieved August 19, 2014, from http://linkedlearning.org/about/evidence-of-effectiveness/

Little, J. W. (2006). Professional community and professional development in the learning-centered school. Washington, DC: National Education Association. Retrieved from http://beta.nea.org/assets/docs/HE/mf_pdreport.pdf

Little, J. W., Gearhart, M., Curry, M., & Kafka, J. (2003). Looking at student work for teacher learning, teacher community, and school reform. Phi Delta Kappan, 85(3), 184–192.


Public Works

Lu, C. M., Kang, S., Huang, S.-C., & Black, J. B. (2011). Building student understanding and interest in science through embodied experiences with LEGO Robotics. In Proceedings of World Conference on Educational Media and Technology 2011 (Vol. 2011, pp. 2225–2232). Retrieved from http://www.editlib.org/p/38169/

Mason, S. A. (2003). Learning from data: The role of professional learning communities. In Annual Meeting of the American Educational Research Association, Chicago, IL.

Mataric, M. J., Koenig, N., & Feil-Seifer, D. (2007). Materials for enabling hands-on robotics and STEM education. Presented at the AAAI Spring Symposium, Association for the Advancement of Artificial Intelligence. Retrieved from http://www.aaai.org/Papers/Symposia/Spring/2007/SS-07-09/SS07-09-022.pdf

Maxwell, L. A. (2013, June 28). New Science Standards Designed for Wide Range of Learners. Education Week, (online). Retrieved from http://www.edweek.org/ew/articles/2013/06/28/36science_ep.h32.html?tkn=UPSF2vS7PKBtE5wq0%2FOvRIL74SkXj7kA%2BOOS&print=1

McREL. (2005). McREL insights: Professional development analysis. Aurora, CO: P.A. Lauer & R. Snow-Renner. Retrieved from http://www.mcrel.org/products-and-services/products/product-listing/01_99/product-68

Meece, J. L., Anderman, E. M., & Anderman, L. H. (2006). Classroom goal structure, student motivation, and academic achievement. Annual Review of Psychology, 57(1), 487–503. doi:10.1146/annurev.psych.56.091103.070258

Mishra, P., & Koehler, M. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. The Teachers College Record, 108(6), 1017–1054.

National Academy Foundation (NAF). (2012). About NAF. Retrieved from http://naf.org/about-naf

National Academy Foundation (NAF). (2014). NAF 2014 Standards. New York, NY: National Academy Foundation. Retrieved from http://naf.org/files/standards2014.pdf

National Academy of Sciences (NAS). (2005). Rising above the gathering storm: Energizing and employing America for a brighter economic future. Arlington, VA: National Academy of Sciences, National Science Foundation (NSF). Retrieved from https://www.nsf.gov/attachments/105652/public/NAS-Gathering-Storm-11463.pdf

National Science Foundation (NSF). (2014). Timeline of NSF History. Retrieved from http://www.nsf.gov/news/special_reports/history-nsf/timeline/index.jsp

NGSS Lead States. (2013). Next generation science standards: For States, by States. Washington, DC: The National Academies Press.

NGSS Lead States. (2013). Next generation science standards: For States, by States. Washington, DC: The National Academies Press.

Nuangchalerm, P. (2011). In-service science teachers’ pedagogical content knowledge. Studies in Sociology of Science, 2(2), 33–37.

O’Driscoll, P. (2000). Documenting the effectiveness of school-to-career strategies: What have we learned?. Pasadena, CA: Public Works.

Palinscar, A. S. (1998). Social constructivist perspectives on teaching and learning. Annual Review Psychology, 49, 345–375.

Pellegrino, J. W. (2013). Proficiency in science: Assessment challenges and opportunities. Science, 340(6130), 320–323. doi:10.1126/science.1232065

Perry, R. R., & Lewis, C. C. (2008). What is successful adaptation of lesson study in the US? Journal of Educational Change, 10(4), 365–391. doi:10.1007/s10833-008-9069-7



Pianta, R. C. (2011). Teaching children well: New evidence-based approaches to teacher professional development and training. Center for American Progress. Retrieved from http://files.eric.ed.gov/fulltext/ED535637.pdf

Pike, G. R., & Robbins, K. (2014). Using propensity scores to evaluate education programs. Presented at the Annual meeting of the Indiana Association for Institutional Research, Indianapolis, IN: Project Lead the Way. Retrieved from education.iupui.edu

Pintrich, P. R. (2002). The role of metacognitive knowledge in learning, teaching, and assessing. Theory into Practice, 41(4), 219–225.

Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33.

Powell, A. (2007, October 11). How Sputnik changed U.S. education. Harvard Gazette, National & World Affairs (Education). Retrieved from http://news.harvard.edu/gazette/story/2007/10/how-sputnik-changed-u-s-education/

Project Lead The Way (PLTW). (2014a). About Us. Retrieved from https://www.pltw.org/

Project Lead The Way (PLTW). (2014b). Our Network. Retrieved from https://www.pltw.org/our-network/college-and-university-partners

Project Lead The Way (PLTW). (2014c). Our Programs. Retrieved from https://www.pltw.org/our-programs

Quillen, I. (2014, January 29). Educators, researchers look for lessons in blended learning algebra program. Education Week. Retrieved from http://www.edweek.org/ew/articles/2014/01/29/19el-math.h33.html

Reiser, B. J. (2013). What professional development strategies are needed for successful implementation of the Next Generation Science Standards? Retrieved from http://www.k12center.org/rsc/pdf/reiser.pdf

Rodgers, C. (2002). Defining reflection another look at John Dewey and reflective thinking. Teachers College Record, 104(4), 842–866.

Roschelle, J. M., Pea, R. D., Hoadley, C. M., Gordin, D. N., & Means, B. M. (2000). Changing how and what children learn in school with computer-based technologies. The Future of Children, 76–101.

Rothman, R. (2011). Something in common: The Common Core Standards and the next chapter in American education. Cambridge, MA: Harvard Education Press.

Sanders, M. (2009). STEM, STEM Education, STEMania. The Technology Teacher, (December/January), 20–26.

Savery, J. R. (2006). Overview of problem-based learning: Definitions and distinctions. Interdisciplinary Journal of Problem-Based Learning, 1(1). doi:10.7771/1541-5015.1002

Schleppegrell, M. J. (2007). The linguistic challenges of mathematics teaching and learning: A research review. Reading & Writing Quarterly, 23(2), 139–159. doi:10.1080/10573560601158461

Schoenfeld, A. H., & Kilpatrick, J. (2013). A US perspective on the implementation of inquiry-based learning in mathematics. ZDM, 45(6), 901-909.

Schraw, G., Crippen, K. J., & Hartley, K. (2006). Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education, 36(1-2), 111–139. doi:10.1007/s11165-005-3917-8

Schunk, D. H. (1984). Self-efficacy perspective on achievement behavior. Educational Psychologist, 19(1), 48–58.

Schunk, D. H. (1985). Self-efficacy and classroom learning. Psychology in the Schools, 22(2), 208–223.


Public Works

Schunk, D. H. (1989). Self-efficacy and cognitive skill learning. Research on Motivation in Education, 3, 13–44.

Scott, C. (2012). An investigation of Science, Technology, Engineering and Mathematics (STEM) focused high schools in the U.S. Journal of STEM Education: Innovations and Research, 13(5), 30–39.

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14.

Siegel-Hawley, G., & Frankenberg, E. (2012). Reviving magnet schools: Strengthening a successful choice option (p. 27). The Civil Rights Project, UCLA. Retrieved from http://civilrightsproject.ucla.edu/research/k-12-education/integration-and-diversity/reviving-magnet-schools-strengthening-a-successful-choice-option/MSAPbrief-02-02-12.pdf

Smarter Balanced Assessment Consortium. (2012). Smarter Balanced Assessments. Retrieved August 1, 2014, from http://www.smarterbalanced.org/smarter-balanced-assessments/

Sparks, D., & Loucks-Horsley, S. (1989). Five models of staff development for teachers. Journal of Staff Development, 10(4), 40–57.

Staub, F. C., West, L., & Bickel, D. D. (2003). What is content-focused coaching. In L. West & F.C. Staub (Eds.) Content-Focused Coaching: Transforming Mathematics Lessons (pp. 1–17).

Stein, M. K., Engle, R. A., Smith, M. S., & Hughes, E. K. (2008). Orchestrating productive mathematical discussions: Five practices for helping teachers move beyond show and tell. Mathematical Thinking and Learning, 10(4), 313–340.

Stine, D. D. (2008). America COMPETES Act: Programs, funding, and selected issues. Washington, DC: Congressional Research Service, Library of Congress. Retrieved from http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA488791

Stone, E. M. (2014). Guiding students to develop an understanding of scientific inquiry: A science skills approach to instruction and assessment. Cell Biology Education, 13(1), 90–101. doi:10.1187/cbe-12-11-0198

Subotnik, R. F., Tai, R. H., Rickoff, R., & Almarode, J. (2009). Specialized public high schools of science, mathematics, and technology and the STEM pipeline: What do we know now and what will we know in 5 years? Roeper Review, 32(1), 7–16. doi:10.1080/02783190903386553

Supovitz, J. A., & Turner, H. M. (2000). The effects of professional development on science teaching practices and classroom culture. Journal of Research in Science Teaching, 37(9), 963–980.

Tai, R. H. (2012). An examination of the research literature on Project Lead the Way. Project Lead The Way. Retrieved from https://www.pltw.org/sites/default/files/PLTW%20DR.TAI%20-%20brochure_pages.pdf

Tai, R. H., Liu, C. Q., Maltese, A. V., & Fan, X. (2006). Planning early for careers in science. Science, 312(5777), 1143–1144. doi:10.1126/science.1128690

The Center for Education Reform. (2014). K-12 Facts. Retrieved from https://www.edreform.com/2012/04/k-12-facts/

The College Board. (2012). AP STEM Access Program FAQs. Retrieved from http://apcentral.collegeboard.com/apc/html/ap-stem-access-program/ap-stem-access-program.html

The White House. (2014a). The White House Science Fair. Retrieved from http://www.whitehouse.gov/science-fair

The White House. (2014b). Maker Faire. Retrieved from http://www.whitehouse.gov/nation-of-makers

Tseng, K.-H., Chang, C.-C., Lou, S.-J., & Chen, W.-P. (2013). Attitudes towards science, technology, engineering and mathematics (STEM) in a project-based learning (PjBL) environment. International Journal of Technology & Design Education, 23(1), 87–102.



Tufts University, Center for Engineering Education and Outreach (CEEO). (2013). Science through LEGO engineering. Retrieved from http://www.legoengineering.com/science-through-lego-engineering/

U.S. Department of Education (USDOE). (1983). A nation at risk: The imperative for educational reform (Evaluative Reports; Policy Guidance No. 065-000-00177-2). Washington DC: U.S. Department of Education. Retrieved from https://www2.ed.gov/pubs/NatAtRisk/risk.html

U.S. Department of Education (USDOE). (2007). A guide to education and no child left behind. Washington, DC: U.S. Department of Education. Retrieved from http://www2.ed.gov/nclb/overview/intro/guide/index.html

U.S. Department of Education (USDOE). (2008). Implementation study of smaller learning communities: Final report. Washington, DC: U.S. Department of Education. Retrieved from http://www2.ed.gov/rschstat/eval/other/small-communities/highlights.pdf

Van Overschelde, J. P. (2013). Project Lead The Way students more prepared for higher education. Project Lead The Way. Retrieved from http://www.utaustinerc.org/files/publications/Project_Lead_the_Way_02_2013.pdf

Vander Ark, T. (2013). The future of learning: Personalized, adaptive, and competency-based. DreamBox Learning.

VEX Robotics. (2012). Curriculum Overview. Retrieved from http://curriculum.vexrobotics.com/curriculum-overview

Vygotsky, L. (1978). Interaction between learning and development. In Mind and Society (pp. 79–91). Cambridge, MA: Harvard University Press.

Waldrip, D. (2013). A brief history of magnet schools. Retrieved from http://www.magnet.edu/resources/msa-history

Yackel, E. (2001). Explanation, justification and argumentation in mathematics classrooms. In PME CONFERENCE (Vol. 1, pp. 1–9). Retrieved from http://didmat.dima.unige.it/miur/miur_dima/G/STORIA_DI_UNA_RICERCA/YACKEL.PDF

California Mathematics & Science Partnership (CaMSP ... · PDF fileCalifornia Mathematics &...

Documents

Transcript of California Mathematics & Science Partnership (CaMSP ... · PDF fileCalifornia Mathematics &...