When tens of thousands of 15-year-olds worldwide sit down at computers to take the Program for International Student Assessment (PISA) examination this fall, they will be tested on reading, math and science. They will also tackle a new and controversial series of questions designed to measure “collaborative problem solving skills.” Instead of short-answer questions or lengthier explanations, the test taker will record outcomes of games, solve jigsaw puzzles and perform experiments with the help of a virtual partner that the test taker can communicate with by typing in a chat box. Although the new test domain is still experimental, PISA officials believe the results from these novel problems will push governments to better equip their young people to thrive in the global economy. Critics of the unit say that PISA has stepped backward into an old and acrimonious debate about whether skills such as critical thinking and collaboration are teachable skills and whether they can be taught independent of content. Given the pace of technological innovation, schools must adapt, and the new domain gives schools a road map to do that, says Jenny Bradshaw, senior PISA project manager, who oversees the test: “Working with unseen partners, especially online, will become a bedrock skill for career success. Increasingly, this is the way the workplace and the world will function.” It is a departure for the 15-year-old exam, which is coordinated by the Organization for Economic Co-operation and Development (OECD), a coalition of 34 member countries guided by industry. Since it was rolled out in 2000, the PISA exam has measured a student’s ability to use reading, math and science in real-life settings. The PISA rankings and the headlines they generate quickly became a flashpoint for policy makers concerned about international competitiveness. The PISA score ranking has fueled, at least in part, a patchwork of efforts at school reform in the U.S. and Europe. America’s mediocre performance on the PISA helped to prompt President Barack Obama to vow in 2009 that U.S. students must “move from the middle to the top of the pack in science and math” within a decade. In 2008 tech industry giants Cisco, Intel and Microsoft, concerned that the job applicants they were seeing were poorly prepared for crucial tasks, began funding their own research through a group called Assessment & Teaching of 21st Century Skills (ATC21S) to identify and promote so-called 21st-century skills—roughly the ability to think critically and creatively, to work cooperatively, and to adapt to the evolving use of technology in business and society. Over several years ATC21S persuaded PISA to begin testing students across the globe for some of these abilities—and found academics to provide a research framework for how this might be done. Three years ago the PISA exam added questions that were supposed to ferret out the problem-solving abilities of 15-year-olds around the globe. (PISA says Chinese students are good problem solvers. Israelis, not so much. Americans fall somewhere in the middle.) A wired, global economy, the test framers decided, requires an even more specific set of skills—group problem solving mediated by the Internet. This year PISA will have students in 51 countries put collaborative problem solving under the microscope. The test questions themselves are alternately fun and frustrating. Although researchers at ATC21S believe it is best to test collaborative problems through actual collaboration, PISA test takers will be paired with a virtual partner dubbed “Abby.” Together the test taker and Abby will be expected, for example, to determine the prime conditions for fish living in an aquarium when the tester controls water, scenery and lighting and Abby controls food, fish population and temperature. To solve the task, the student must build consensus around how to solve the problem, respond to concerns, clear up misunderstandings, share information from trials and synthesize the results to come up with the correct answer. Plenty of critics say the new domains are a blunder. “Is there an independent set of skills—in this case, collaborative problem solving—that is transferable across domains of knowledge?” asks Tom Loveless, an education researcher at the Brookings Institution. “Is problem solving between two biologists the same as problem solving between two historians? Or is it different? Progressive educators since John Dewey have insisted it is the same, but we just don’t know that.” School systems that want to prepare students for the future should help them achieve mastery of complex math, science and literacy instead of putting resources into promoting nebulous concepts. PISA’s Bradshaw acknowledges that questions do remain about the innovative domains but that she and her team believe it is an experiment worth trying. While PISA researchers conduct validation studies and focus groups on collaborative problem solving, others are already working on PISA’s next frontier. By 2018 she says her team will have come up with a valid way to measure “global competence.” Because it is true in education that what gets tested gets taught, ATC21S is preparing for the international hand-wringing from low-ranked countries by offering videos of classrooms where the researchers say teachers and students are getting it right. It has also rolled out a MOOC (massive open online course) to train teachers how to bring collaborative problem solving into their classrooms; 30,000 teachers have enrolled in the course, and a quarter of them have completed it.
Critics of the unit say that PISA has stepped backward into an old and acrimonious debate about whether skills such as critical thinking and collaboration are teachable skills and whether they can be taught independent of content.
Given the pace of technological innovation, schools must adapt, and the new domain gives schools a road map to do that, says Jenny Bradshaw, senior PISA project manager, who oversees the test: “Working with unseen partners, especially online, will become a bedrock skill for career success. Increasingly, this is the way the workplace and the world will function.”
It is a departure for the 15-year-old exam, which is coordinated by the Organization for Economic Co-operation and Development (OECD), a coalition of 34 member countries guided by industry. Since it was rolled out in 2000, the PISA exam has measured a student’s ability to use reading, math and science in real-life settings. The PISA rankings and the headlines they generate quickly became a flashpoint for policy makers concerned about international competitiveness. The PISA score ranking has fueled, at least in part, a patchwork of efforts at school reform in the U.S. and Europe. America’s mediocre performance on the PISA helped to prompt President Barack Obama to vow in 2009 that U.S. students must “move from the middle to the top of the pack in science and math” within a decade.
In 2008 tech industry giants Cisco, Intel and Microsoft, concerned that the job applicants they were seeing were poorly prepared for crucial tasks, began funding their own research through a group called Assessment & Teaching of 21st Century Skills (ATC21S) to identify and promote so-called 21st-century skills—roughly the ability to think critically and creatively, to work cooperatively, and to adapt to the evolving use of technology in business and society. Over several years ATC21S persuaded PISA to begin testing students across the globe for some of these abilities—and found academics to provide a research framework for how this might be done.
Three years ago the PISA exam added questions that were supposed to ferret out the problem-solving abilities of 15-year-olds around the globe. (PISA says Chinese students are good problem solvers. Israelis, not so much. Americans fall somewhere in the middle.) A wired, global economy, the test framers decided, requires an even more specific set of skills—group problem solving mediated by the Internet. This year PISA will have students in 51 countries put collaborative problem solving under the microscope.
The test questions themselves are alternately fun and frustrating. Although researchers at ATC21S believe it is best to test collaborative problems through actual collaboration, PISA test takers will be paired with a virtual partner dubbed “Abby.” Together the test taker and Abby will be expected, for example, to determine the prime conditions for fish living in an aquarium when the tester controls water, scenery and lighting and Abby controls food, fish population and temperature. To solve the task, the student must build consensus around how to solve the problem, respond to concerns, clear up misunderstandings, share information from trials and synthesize the results to come up with the correct answer.
Plenty of critics say the new domains are a blunder. “Is there an independent set of skills—in this case, collaborative problem solving—that is transferable across domains of knowledge?” asks Tom Loveless, an education researcher at the Brookings Institution. “Is problem solving between two biologists the same as problem solving between two historians? Or is it different? Progressive educators since John Dewey have insisted it is the same, but we just don’t know that.”
School systems that want to prepare students for the future should help them achieve mastery of complex math, science and literacy instead of putting resources into promoting nebulous concepts.
PISA’s Bradshaw acknowledges that questions do remain about the innovative domains but that she and her team believe it is an experiment worth trying. While PISA researchers conduct validation studies and focus groups on collaborative problem solving, others are already working on PISA’s next frontier. By 2018 she says her team will have come up with a valid way to measure “global competence.”
Because it is true in education that what gets tested gets taught, ATC21S is preparing for the international hand-wringing from low-ranked countries by offering videos of classrooms where the researchers say teachers and students are getting it right. It has also rolled out a MOOC (massive open online course) to train teachers how to bring collaborative problem solving into their classrooms; 30,000 teachers have enrolled in the course, and a quarter of them have completed it.
