Why TIPP&SEE Works: CANON Lab Research Wins Best Paper at CS Education Conference

In education research, it’s one thing to show that a learning approach works, but a whole lot harder to understand why. Earlier this year, PhD student Jean Salac and researchers from the CANON Lab directed by associate professor Diana Franklin found that TIPP&SEE, a computer science learning strategy that draws inspiration from reading comprehension, was effective in helping upper elementary students learn and apply basic computing concepts. In their follow-up, the team went under the hood to try to figure out what made TIPP&SEE tick.

Their efforts yielded some surprising results and a Best Paper award at the 2020 International Computing Education Research (ICER) conference, the most prestigious ACM conference on computer science education research. Conference reviewers rewarded the researchers for blending cognitive science and learning science to study what made TIPP&SEE effective in its pilot trial, digging into the “artifacts” of the original study to guide future expansions of the strategy. 

“The original SIGCSE paper provides early evidence that TIPP&SEE is effective. The purpose of the ICER paper is to gain insight into why it might be effective,” said Prof. Franklin.

TIPP&SEE works with the Scratch introductory programming language that is commonly used for early computer science education. But the TIPP&SEE steps build upon that curriculum, inspired by strategies used to teach young children to read and organized around a central protocol of “Use →  Modify → Create.” Students preview a project, look at the code that makes it run, then tinker with the instructions to see how it affects the program. In the final step, they take what they have learned and attempt to build their own code using those principles. 

The original study, conducted in 16 classrooms of students aged 9 and 10, found that students taught with TIPP&SEE exhibited better understanding of computational concepts such as events, sequences, and loops than control students. But that paper, presented at the 2020 SIGCSE meeting in March, only looked at final assessments. For the new ICER study, teams of undergraduates (Zack Crenshaw, Saranya Turimella, Zipporah Klain, and Marco Anaya) over two quarters analyzed additional data collected during the experiment, including worksheets completed by the students and the programming projects they completed.

By studying these “artifacts,” the researchers could gain insight into whether the metacognitive approach of TIPP&SEE successfully promoted behaviors that lead to learning in computer science. 

“Metacognition is the process of thinking about your own thinking,” Salac said. “Think about the steps you take to read an article. As you're reading, you're breaking down the paragraphs into sentences, and then sentences into words and as you assign meaning to individual words, you build up to an understanding of the full sentence. All of this happens quickly and implicitly in an adult reader's head, but to a beginner, these steps can be overwhelming. A similar process happens with reading code, and strategies like TIPP&SEE help make these implicit processes used by experts explicit so beginners can better learn the overarching task.”

When the team looked at student performance on individual projects, they found several encouraging signs that the intervention was inspiring this metacognitive process. Students in TIPP&SEE classrooms completed more project requirements, used more programming elements, and created longer scripts.

However, when the researchers looked at the worksheets that TIPP&SEE students were assigned to complete during instruction, they found a surprising result — or lack thereof. Students reliably completed the Use and Modify sections of the worksheets, but a majority failed to record answers to the third (and what researchers originally thought might be the  most important) step of the protocol, Explore. In addition, there were no correlations between assessments, project attributes, and worksheets. 

“We were very surprised because we saw that the TIPP&SEE students were doing so well with their project requirements,” Salac said. “But we gained some insight when we talked to some of the students — when it came down to it, some students just didn't have time to finish everything. Or maybe they just didn't write it down; they got too busy exploring. But that's something that we have to observe more, to see what students are doing during the projects.”

Despite the lack of correlation, the result will help Salac, Franklin, and CANON Lab design the next phase of TIPP&SEE, for an upcoming study that will expand to multiple school districts. By redesigning class materials and finding new tools for tracking how students work through projects, the next phase can gain deeper understanding on how the learning strategy sticks with students and provides a pathway to computational thinking.

In addition to the award-winning paper, the CANON Lab also presented two additional papers at ICER ‘20 (out of a total of 27 papers). In “An Analysis of Use-Modify-Create Pedagogical Approach’s Success in Balancing Structure and Student Agency,” Franklin and collaborators test whether the protocol at the heart of TIPP&SEE provides a balance of structured and open-ended exploration. In “Exploring Quantum Reversibility with Young Learners,” Franklin’s team studied how to teach a basic concept in quantum science to 8- and 9-year-old students.