- Get Involved!
A Guide to Conducting Educational Research in the Planetarium
Julia D. Plummer1, Shannon Schmoll2, Ka Chun Yu3, and Chrysta Ghent1
1 Department of Curriculum and Instruction, Pennsylvania State University, firstname.lastname@example.org, email@example.com
2 Abrams Planetarium, Michigan State Universe firstname.lastname@example.org
3 Denver Museum of Nature & Science email@example.com
Published in Planetarian, Vol. 44, No. 2, June 2015 pages 8-24, 30
For the past several decades, researchers have conducted studies on the planetarium as an educational venue. A major goal of this research has been to measure student conceptual learning (e.g. Brazell & Espinoza, 2009), while other studies have compared learning outcomes in the planetarium to learning in other settings (e.g. Zimmerman, Spillane, Reiff, & Sumners, 2014). Despite the long history of research on the planetarium, there are many questions unanswered about the role planetariums can play in educating audiences or supporting their interest in astronomy.
The goal of this article is to provide guidance to the planetarium community on issues pertaining to planetarium-based research.
This guide is organized into four sections. First, we present four broad areas of literature in which planetarium-based research could be situated. Second, we discuss basic elements of conducting educational research and different approaches to collecting and analyzing data. Third, we suggest practical considerations that are important to remember when engaging in planetarium-based research. Finally, we discuss the importance of disseminating research results and identifying potential venues in which to present and publish. We see this guide as one part of an ongoing conversation that can lead to improvement in how the planetarium community implements research-based programming for our audiences.
Potential research areas and related literature
In this section, we first explore the spectrum of informal and formal learning within education literature and examine the planetarium’s place in this body of work. Next, we discuss the planetarium-specific spectrum of learning, in and out of the dome, and why the planetarium community should consider both of these spaces when building a research literature base. In considering the intersection of these realms, we discuss four potential lines of research as examples of the kind of inquiry members of the planetarium community could pursue. This will allow us to introduce some relevant literature, but will not act as an exhaustive analysis of existing work or potential lines of research.
Informal vs formal learning environments
The terms formal and informal learning environments are often applied to the dichotomy of in school and out of school contexts, respectively. However, there are a variety of characteristics that distinguish informal and formal learning environments that will help us define the planetarium learning experience in more nuanced ways. A primary characteristic of informal learning environments, such as museums, science centers, and nature centers, is the level of choice offered a learner (Falk & Dierking, 2000). On the other hand, formal learning environments are often characterized by a teacher-led curriculum where students have little control over what they study.
Thus, informal environments differ from formal environments by the extent to which learners’ experiences are driven by their choice in which spaces to visit and control over how long to linger, resulting in more personalized experiences (NRC, 2007). People also spend different amounts of time in informal environments, usually with less frequent visits that are shorter than a typical classroom experience. Additionally, there are documented “novelty” effects which occur when new or infrequently visited spaces cognitively overwhelm people; this makes it difficult for them to learn content because they are distracted by what is new and different about the environment (e.g. Orion & Hofstein, 1994).
As a result, many informal spaces tend to focus their learning goals on affective gains, instead of learning factual content. These emotional aspects of learning include motivation, interest, and perseverance to learn something new. The focus on meeting standards in formal education means that the content and cognitive gains are often given a higher priority.
Another difference between informal and formal environments is primary audience. Schools and colleges focus on students who are required or pay to attend, and receive individual grades. Informal museums, on the other hand, cater to the wider public audience who may view their visits as social excursions.
While these differences in how we consider learners’ experiences do exist between formal and informal learning environments, they should not be considered a stark dichotomy, but as a fluid spectrum. Every learning experience will likely fall somewhere between these extremes (NRC, 2009). For instance, formal classrooms may emphasize choice and control by allowing students to pursue their own research projects, albeit in a certain content area. Similarly, people may choose to visit a museum and follow a more structured docent-led tour. Where exactly a learning environment or an individual educational program falls will depend on content, audience, physical space, and overall goals of both the educators and the audience.
Programs at individual planetariums will also fall at different points on this spectrum. planetariums housed in schools may implement programs that fall more in the formal realm, as it could be considered a specialized classroom that students can return to frequently. Those that operate as part of museums or science centers may fall in the informal realm as they cater to wider audiences who choose to spend their leisure time attending a planetarium program. Portable domes that travel to a school might fall closer to the middle as they are single visits, but also may be integrated into a school curriculum. Similarly, some planetarium programming that is more structured and show-like might not offer a lot of choice and social interaction, thus not connecting to the same ways of learning as we consider to happen in informal environments. Planetarium shows that are designed to offer the audience more control might fall on the informal side.
Understanding these different characteristics of formal and informal learning environments will help you start reviewing appropriate research literature for a new research project. In other words, even though planetariums are often categorized as a form of informal education, the nature of how we engage our audiences suggests that drawing on literature from formal, school-based research rather than from the museum-based literature may also be a productive choice.
In-dome vs out-of-dome
Even though existing informal and formal literature can inform our work, we still need separate evidence-based planetarium research because of the unique characteristics of planetariums compared to other learning environments. For example, school-based planetariums often provide an educational experience more akin to an informal environment when compared to the rest of their school-based curricula. This means that school-based planetarium educators may need to attend more to the novelty factor of the planetarium environment while also benefiting from opportunities to engage children’s excitement and interest. When located in an informal environment, planetariums offer more structured learning activities and time constraints than other parts of a museum or science center, skewing the planetarium experience back towards a formal education experience. Having evidence unique to our range of audiences and contexts will help us better identify best practices for the field.
And, despite our unique environments, we should not restrict ourselves to only what happens in the dome. People do not come into a learning environment as blank slates; rather they construct knowledge in different ways based on prior experiences and ideas (Piaget, 1970). Additionally, it has long been recognized that there are important social factors to how people learn. Learning occurs best through interactions between people (Vygotsky, 1978). This is something that informal learning environments can easily foster (Falk & Dierking, 2000).
For planetariums, the social piece is often lacking because talking during a show violates social etiquette and time is limited for any social interaction. If interaction during a show happens, it is often still passive and limited to visitors answering a question posed by a presenter or voting on something, like a topic or destination, by the audience. As a result those social aspects of learning will more likely happen, and may need to be fostered, beyond the dome.
Research should investigate how these out-of-dome experiences could inform what we do in the dome and how the dome can influence extended learning after a presentation. Because what happens outside this dome affects learning in the dome, we consider this a separate spectrum of learning in and out of the dome.
Figure 1: Quadrants of Planetarium Research (click on image to see larger version)
The intersection of informal, formal, in-dome, and out-of-dome creates what we will refer to Quadrants of Planetarium Research (see Figure 1). This is not the only way we can split research, but the quadrants could be a helpful tool to situate your project in the literature and develop appropriate research questions. We will present vignettes of potential lines of research in each quadrant as a way to illustrate potential avenues for future research.
Formal and in-dome quadrant: Effective visualizations for multiple audiences
The heart of our work in the planetarium is the way we visually engage audiences. And yet, what do we know about how our audiences interpret and extract information from the visualizations used in the dome, whether we project a simple star field with constellation overlays or complex renderings of flight through the solar system?
Within this area of research exists many potential lines of inquiry focused on the cognitive gains associated with visualization. As a result of this conceptual and cognitive emphasis in the dome this particular visualization research would fall into the formal/in-dome quadrant.
Some existing research has begun to explore ways an immersive planetarium environment may support audiences in making sense of spatial information. Chastenay (2015) investigated ways a planetarium program can be designed to support both an Earth-based and space-based perspective on the lunar phases. Yu, Sahami, Sahami, and Sessions (2015, in press) have begun to uncover evidence that visualizations presented in an immersive, fulldome environment are better at supporting student learning of spatially complex phenomena than the same visualization on a non-immersive (flat) format. This may be due in part to the immersive visuals freeing up cognitive resources that would otherwise be used to keep track of where celestial bodies are in the simulation.
Further research in this area may need to draw upon the existing body of multimedia theory literature (Mayer, 2005). Considerable prior research on multimedia education points to the importance of considering the cognitive load incurred by visuals or animations due to limited size of human’s working memory. One consequence of this is the split-attention effect (Sweller, Merrienboer, & Paas, 1998): if the visualization provides multiple sources of visual information, all of which are essential for understanding the content presented, then the learner must integrate this information internally before making sense of the visualization, which imposes a heavy cognitive load.
Gillette (2014) found a negative impact on student learning from planetarium programs that include additional images or deviations from the main content presentation, which further supports the importance of attending to multimedia theory. An important instructional implication of this is that researchers and program developers should consider the types of visual elements presented and find better ways to effectively integrate them for the learning in the planetarium presentation.
This is but one direction research might take on the design of effective dome visualizations. Other lines of research might consider what makes a visualization for a particular concept or age group effective by examining how it supports audience’s ability to engage with spatially complex scenarios. Additional research could consider the relationship between students’ prior conceptual knowledge or spatial skills and what they learn from particular dome visualizations. Other studies could explore methods of educating audiences in the practices by which astronomers investigate the universe. Overall, future studies in this area should carefully consider the nature of how planetariums engage audiences, as well as existing literature on how people learn from images and simulations.
Formal and out-of-dome quadrant: Intersections and connections between planetariums and school-based education
Field trips for students represent a large portion of many planetariums’ audience. While studies have investigated student learning during field trips to planetariums (e.g. Plummer, 2009), much more could be done to consider methods that further support this population across both learning environments. Prior research on student learning on field trips to museums and other informal settings has concluded that student learning is improved when the field trip is integrated into students’ school curricula (Griffin, 1998, Dewitt and Osborne, 2007). This generally takes the form of classroom-based pre- and post-activities that are directly tied to what students see while on the field trip.
Because integration includes out-of-dome preparation and follow-up in the traditionally formal realm of classroom education, this line of research would fall within the Formal/Out-of-Dome quadrant. However, given current understanding of how learning is tied to the context in which it is learned (e.g. Brown, Collins, & Duguid, 1989), more research is needed on effective ways of integrating students’ experiences across informal and formal learning environments.
Some research has already been conducted in this area, though it is a burgeoning field for planetariums. One approach to this problem was Schmoll’s (2013) study of how the School-Museum Integrated Learning Experience in Science framework (Griffin, 1998) could be used to effectively integrate a school astronomy unit with a planetarium visit. Another line of investigation could explore methods of designing curriculum supplements aligned to specific planetarium programs. For example, Small and Plummer (2014a; 2014b) investigated the impact of a combination of planetarium field trip and a pair of pre/post classroom lessons, designed to facilitate further engagement with the planetarium content, on early elementary students’ understanding of lunar phenomena.
The work of Schmoll (2013) or Small and Plummer (2014a; 2014b) could be expanded with different planetarium types, age groups, or content areas. Schmoll (2013) also noted that more research needs to be done to explore how to best foster the social aspect of learning in and out of dome. We can also look at understanding the role that teachers play in these field trips to understand how we can best collaborate with and support them in this integration. We could also look at similar research done already in other informal environments such as museums to gain an idea of other research questions that should be explored in planetariums.
Informal and in-dome: Choice in the planetarium
A third broad area of potential research could focus on audience experiences in the planetarium through opportunities for free-choice learning. Because choice is such a key feature of informal environments, this would fall into the informal/in-dome quadrant. It can be difficult to incorporate choice into planetarium shows, but not impossible. However, audience members’ opportunities for choice under the dome will likely look different from choice in other aspects of a museum, such as walking through a gallery space. While many planetariums include some opportunities for live engagement, the level of controlled facilitation in planetariums limits the extent to which visitors can personalize their experience.
Some planetariums offer choice and interaction through clickers or a similar kind of voting system, though little about the effectiveness of these interactions towards increasing audience learning or interest has been formally published. Given the importance of visitors’ own personal interests and motivation in what they take away from an informal science setting (Falk et al. 2006), are there particular ways in which planetariums could support effectively choice in the dome or is it best to introduce choice through experiences beyond the dome? Future research in planetariums settings could further investigate how this theoretical framework can inform the design of planetarium experiences and lead to more research on the relationship between visitors’ personal agendas and the design of planetarium programs.
Informal and out-of-dome: Extending learning opportunities out of the dome
A final area for research considers the potential for connecting visitor experiences in the dome to additional opportunities for sense-making and exploration outside of the dome. In the previous section, we raise the possibility that visitor agendas and the potential for personal choice could lead to innovations in research within the dome. Similarly, in this quadrant, we suggest that visitor experiences in the space outside the dome provides a wealth of opportunity to pursue their interests and discuss their planetarium experiences with family members. This is similar to the literature on integrating a field trip into curriculum through pre- and post-activities. However, there is a shift in audience and additional level of choice that moves this line of research into the informal side of the spectrum, while remaining Out-of-Dome.
One line of research could explore methods of connecting temporary or permanent exhibit displays to planetarium content. Extensive research has considered methods of optimizing visitor engagement with museum exhibits (e.g. Allen, 2004), but little has been done to explore how to connect planetarium experiences with other aspects of a museum visit. Some research has explored how visitors make their own choices and decisions about how various exhibits are connected within a museum (MacDonald, 2007), but how could a museum effectively engage visitors in making purposeful connections between a planetarium visit and other exhibit space, especially considering the differences in the nature of these experiences? And what ways might this shape a visitors’ learning, interest, or motivation to engage in future astronomy-based experiences?
These are only a few potential options. Many of these questions are likely to span across informal, formal, in-dome and out-of-dome in ways that go beyond these examples. For instance, we discussed social learning being pushed beyond the dome, but questions could be asked relating to how we can foster social interactions during a show. Issues of choice in the dome could be studied beyond the dome. Questions of classroom-field trip integration could look at extended programming while at the planetarium. There are many lines of worthy inquiry if set up appropriately within the literature and methodology base. To start familiarizing yourself with the current state of the research field, we recommend reading some of the relevant research syntheses published and freely available to download (www.nap.edu) by the National Research Council. We have selected the reports most relevant to inquiry in the four quadrants of planetarium research (See Table 1). In the next section we will discuss how to take the next steps in designing your research study.
Basics of Research
As we review some of the basics of conducting educational research, keep in mind that this will be a brief introduction rather than a comprehensive summary of the topic. There are entire textbooks and primers devoted to this topic (see Table 2 for suggested readings). Our goal in this section is point those new to research in the right direction for deciding on research questions, methodologies, and practical considerations for any study.
Research vs evaluation
One question that often comes up with an investigation that involves specific instructional interventions, such as a planetarium program, is whether the study would be considered research or evaluation. Evaluation and research may seem very similar given the focus on developing evidence-based conclusions and use of the same tools and terminology, there are distinct differences between the two.
The goal of evaluation is to improve a program or educational approach by judging the program itself. The evaluation serves the stakeholders who created or funded the program. There are different types of evaluation, depending on how far along you are on a project. Formative evaluation is performed during program development; results are used to refine the program. Summative evaluation is done at the end to judge the success or failure of a program in achieving its goals.
Research, on the other hand, focuses on questions that go beyond a particular program, with the goal of making discoveries that can be generalized to other programs, audiences, and conditions. Research may be used to test a specific theory or to generate new theories and hypotheses. Instead of gathering data to provide evidence for stakeholders who wish to determine the effectiveness of a specific program, research often originates from scholars, with the merit of the research judged by other scholars from the same discipline. While many of the concepts, methods, and perspectives we discuss in this paper are applicable to either research or evaluation, our focus here will be on addressing questions and practices of research.
For those interested in getting started in planetarium-based research, you may find it productive to start by thinking about ways to conduct research on how you engage your audiences. This would likely be categorized as a third approach referred to as action research. Action research has its roots in classroom-based research wherein teachers engage in research on their own teaching practices in their own classrooms. Those engaged in action research often focus on the immediate application of findings to practice, rather than producing generalizable results or generating theory. Thus, action research is often cyclical with the researcher gathering evidence on outcomes of their instruction and using those results to make improvements in their practice, followed by further data gathering and improvement
At the core of any research study are the questions being asked and answered. Our research questions drive all further considerations of appropriate evidence to gather and methods to use in a study. Yet crafting a good research question is one of the most challenging aspects of doing research (Slater, Slater, & Bailey, 2010; Slater, Slater, & Shaner, 2008). A good research question should connect to a broad base of existing research literature.
The literature should be used to effectively identify areas in which more research is needed. A good question might reframe questions which have been previously asked but apply them to new contexts or use improved data gathering tools. For example, questions about learning or instructional design studied in non-planetarium settings may now be applied to new research conducted in the planetarium environment, or research previously done in one type of planetarium, such as within a small dome or with a optical-mechanical projector, might be done in a larger planetarium or with fulldome projections to better understand how these differences affect learning.
The literature also should be used to provide a rationale for the research question, by showing the significance of the problem being studied and how it draws on previous findings or theories. Good research questions address issues that are meaningful to the community, such as others in the planetarium field, planetarium researchers, and related constituencies.
Finally, good research questions are those that can be answered using evidence. Thus, when selecting a research question, the researcher also should consider the available resources for gathering data. Beginning with a research question that leads to a small-scale pilot study may be the appropriate first step towards broader research questions answered in larger studies in the future.
There are three, interrelated, categories of research questions in education (NRC, 2002): descriptive (What is happening?), causal (Is there a systematic effect?), and mechanistic (How and why is it happening?). Descriptive questions are those that attempt various kinds of descriptions of people, actions, or events. A descriptive, planetarium-relevant research question might include: How often do teachers use content addressed in the planetarium before or after a planetarium visit? Descriptive questions also address simple relationships or correlations between variables when methods do not allow for causal conclusions. An example of this type of research might examine the design of a planetarium program that is meant to support an audience’s understanding of size and scale; the descriptive question might be, do audience members understanding of relative size and scale improve after attending this planetarium program?
Causal questions attempt to control for the potential that an alternative explanation could account for the result by providing a comparison group. In other words, if one is investigating whether a particular educational intervention (planetarium program, exhibit, classroom lesson, etc.) causes the change in audience knowledge or behavior, the audience outcomes should be compared to another group of similar people who did not attend that particular intervention. For example, one might ask: Are audiences’ explanations of [topic] more accurate when they first engage with a related exhibit before the program compared to after the program? Such research could examine participants who visited an exhibit before, after, or not at all.
Finally, mechanistic questions take a closer look at the process and mechanisms by which factors may influence outcomes. For example, if someone conducted the previous study and found that audiences learn more when they are primed by exploring a related exhibit prior to entering the planetarium than if they did not first explore that exhibit, a follow-up study might try to uncover the mechanism that influences that difference: Why does engaging with an exhibit prior to attending the planetarium improve learning outcomes? Such a study may need to examine the social interactions that take place between visitors regarding the exhibit as well as developing methods to investigate differences in what audience members attended to during the program. Findings could then influence future exhibit and planetarium development.
The choice of research question should drive the selection of appropriate research methods. Paying close attention to the match between research question and methods allows for “the development of a logical chain of reasoning based on the interplay among investigative techniques, data, and hypotheses to reach justifiable conclusions” (NRC, 2002, pp. 62-63).
In general, educational researchers divide methods into two broad categories: quantitative and qualitative. Not only do quantitative and qualitative research have different philosophical approaches, but they have different methodologies as well. Each approach has different strengths, which allow researchers to solve different types of problems.
Quantitative research may be most familiar to physical scientists; questions are answered based on the analysis of numerical data. Quantitative methods allow the researcher to measure cause and effect, determine statistically significant changes in variables, and look for correlations between variables. In quantitative studies, researchers first identify variables that may influence learning. Only one variable should changed at a time during a study. Thus, researchers must carefully consider how to control for potentially confounding variables (extraneous variables which may also influence the outcomes). For example, in a study of whether a planetarium program supports student learning, the researcher will need to make sure that other experiences that the audience has outside of the dome does not influence the results of a post-assessment.
Quantitative methods can also be used to generate descriptions of educational phenomena by using descriptive statistics (e.g. how many participants engage in different types of behaviors). Data is often gathered using surveys, tests, and other quantifiable instruments; this allows for large sample sizes and the potential for greater certainty that a particular finding is representative of the population being studied.
Qualitative research, on the other hand, often uses more subjective methods, which rely more on the researchers’ observational and interpretive abilities. Qualitative researchers may collect data through interviews, open-ended questionnaires, field notes, and video analysis. The qualitative researcher assumes that individuals are affected by the world around them, and thus the methods of collecting data should account for these influences.
While quantitative research often works towards testing a preconceived hypothesis, qualitative research methods allow for flexible analyses that help researchers identify unanticipated phenomena or events in ways that can lead to new hypotheses. In other words, quantitative methods are often used when there is sufficient existing literature to form clear hypotheses and develop instruments that can measure variables of interest, while qualitative methods may be more appropriate when the researcher needs to gather more information in order to better understand an educational context or problem that has been insufficiently studied. Using observational and interpretive methods to characterize individuals and situations takes a considerable amount of time, so qualitative research studies tend to involve small sample sizes.
The mixed methods approach is, as the name suggests, a combination of both quantitative and qualitative. For some research questions, combining these two allows the researcher to build on and offset the weakness of the other approach. There are also situations where using only one type of methodology is insufficient to answer the research question.
One approach to using mixed methods might be to collect both quantitative and qualitative data concurrently, such as asking visitors to fill out surveys about their interest in a topic (quantitative) while also taking field notes on how they respond to a particular program (qualitative); the researcher would then merge the results of analyzing each data set to interpret the findings.
An explanatory approach might be to first collect and analyze quantitative data followed by collecting and analyzing additional qualitative data. The goal in this design would be to use the qualitative data, such as purposefully selecting visitors to interview after attending a planetarium program, to explain results measured quantitatively.
An exploratory approach might include gathering extensive qualitative data, such as pre- and post-visit interviews, and then once patterns emerge in participants’ answers, conduct statistical analyses to measure whether those emergent themes or characteristics changed after a visit to the planetarium.
A final approach is called Design-Based Research (DBR; Design-Based Research Collaborative, 2003), also known as design experiments or design-based implementation research. DBR draws on qualitative and mixed methods traditions to study learning environments through an iterative process of refining interventions in order to develop new theories and methods of instruction. The key focus here is on gathering data from cycles of implementation: the researchers and developers continue to revise their practice or programs, based on prior rounds of data collection, in ways that help them better understand how the intervention supports learning.
Studies using DBR focus on understanding learning in naturalistic settings and believe that the value of this approach is that it recognizes and integrates the messiness of real-world learning environments (Barab, 2014). DBR considers that the design of a program is inseparable from the implementation of a program; in other words, rather than treating an educational experience in and around the planetarium as a fixed object, design-based researchers consider that research on any designed program must also consider the larger ecosystem in which it is expected to engage an audience (Barab, 2014). DBR considers the differences encountered based on different audiences, different presenters, and other aspects of the learning environment, as critical to both interpreting and communicating the results of a study.
Examples of research methods
To illustrate the relationship between these methodological approaches and research questions about the planetarium environment, we will consider a single problem space. Within this space, we will describe different types of questions that could be asked and how this would lead to different methodological approaches. Imagine the hypothetical case of a planetarium professional who is working with 5th grade teachers (10-11 years) in a local school district. The planetarium professional has been asked to help the teachers address some of the Next Generation Science Standards (NGSS Lead States, 2013), a new set of standards for science education in the United States, including:
While this is a real, practical problem for the teachers in this school district, it is also a context in which to engage in research that would improve our understanding how best to engage students in planetarium field trips.
Based on the existing literature, the planetarium professional knows that it will be important for the planetarium program she creates to be well integrated in to the teachers’ curriculum (Schmoll, 2013, Griffin, 1998). Several different—yet equally useful and relevant—research questions, with their corresponding methodologies, could be asked about this problem space.
Quantitative example: She could design a planetarium program that helps students make sense of distances to stars or change in our observations over time by providing visualizations that help them make sense of data they had been exploring in the classroom; students would visit the planetarium near the end of their unit as a way to help them construct reasoning for the scientific arguments they are making in the classroom.
A research question might be posed, Do students who attend a planetarium program designed to help them reason about [astronomical phenomena] demonstrate increased learning gains compared to students who only participated in their typical school astronomy unit?
The planetarium researcher could do a quantitative comparison of pre/post gains, using existing multiple choice assessments, from students who attended the planetarium program during their astronomy unit compared to students who participated in the same astronomy unit but did not attend the planetarium. Findings might be used to advocate for future classroom-planetarium partnerships while also adding to our understanding methods that support student learning.
Qualitative example: On the other hand, the researcher might ask a qualitatively-driven research question, such as: In what ways did students who attended a planetarium program designed to help them reason about [astronomical phenomena] use this information when engaging in small group, whole group, and written scientific argumentation when they returned to the classroom?
This question might be answered using qualitative methods that carefully consider student discourse and patterns in their writing to draw conclusions about the interaction between the planetarium program and classroom learning. Findings could be used to develop similar classroom-planetarium programs in the future and consider ways that planetariums can help students engage in evidence-based reasoning in the classroom.
Mixed-methods example: A question which might lead to a mixed-methods study might ask, In what ways do students’ explanations for [astronomical phenomena] change after participating in a planetarium program and associated classroom lessons? Data collected could include open-ended written assessments or interviews conducted before and after instruction. Analysis might use open coding to uncover themes in students’ answers.
These themes could then be rank-ordered according to the accuracy or use of an evidence-based explanation structure, then change could be calculated using statistical measures. The qualitative aspect of coding open-ended questions allows the researcher to uncover student ideas that would be lost using a traditional multiple choice test; however, this type of research is also more labor intensive than a strictly quantitative approach so the trade-offs must be considered.
Design-based research example: Finally, this problem space might be an opportunity to engage in design-based research while answering the question, Can a field trip to a planetarium provide students with scientific models that can be used to engage in scientific argumentation in the classroom? With a DBR approach, the research would be conducted in iterative cycle. During each iteration, data would be collected that would help the researcher understand how all of the participants are working in this system (the planetarium operator, classroom teacher, and the student) and how these interactions relate to learning outcomes.
Findings from multiple sources during the first iteration would be used to revise an initial theory of how to design the integration of the planetarium and classroom instruction before the next implementation. Through multiple cycles of data collection, which might include audio and video of instruction, interviews with key participants, and pre/post assessments, the intervention and underlying assumptions for its design would be revised and improved.
While these potential lines of research began by considering the need to help teachers in the U.S. to address a new set of national science education standards, we encourage planetarium educators in other countries to consider the broader question of supporting student learning by consulting their own local or national standards.
Forming partnerships with other researchers and the communities that you are studying probably will be necessary. Most of us in the planetarium field spend our time creating and delivering programming to our audiences. Few in our community have had the necessary training to develop a research program, so that is where partnering with someone with the quantitative or qualitative research expertise is important.
Do not be discouraged if the first researcher you contact is not interested in or qualified to investigate the research questions you want to ask. She or he may be able to can point you to others who may be better suited to supporting your interests. When starting a potential research collaboration, it is worthwhile for the two of you to explore the problem space you are interested, find out where common interests lie, and collaboratively come up with the research question and investigative approach. While coming in with a specific set of research questions might get a conversation going, be ready to make changes and adapt as your potential collaborator may suggest changes that better reflect the current research literature or would allow for the researcher to also explore her or his own personal research agenda.
If you want to study the impact of your program on students in a local school district, engaging in a dialogue with people in that district will be important for two reasons. First, different schools will have different rules for approving educational research. You may only need to get approval from the principal at some schools, while in other cases you will need administrative approval at the district level or from the school board. All of this requires time, so prepare accordingly; do not start contacting school officials a month before the start of classes! The groundwork for creating a research study should be laid at least a semester if not a full year before you plan to start your study.
Second, regardless of who in the school district has final say for approving your project, it is important to develop relationships with a teacher or with the district curriculum developer. You may already have an idea for a program to test out, but teachers and schools have their own requirements for what must be taught. They are likely to be more aware of what needs to be covered to meet local or national science standards, to prepare students for required standardized tests, and what types of educational interventions will be possible in their local context.
Just like the give-and-take that comes with working with education researchers, partnerships with schools and teachers will allow you to create programming that not only answers your research question, but meets the needs of the school and children, as well.
Institutional Review Boards
U.S. Federal law requires that any research study involving human subjects be reviewed and approved by an Institutional Review Board (IRB). Our experience in this area has been in U.S. contexts and under U.S. laws and thus we will be writing this section from that perspective. However, many other countries also regulate human subjects research, so all researchers must be careful to look into the laws and regulations in their local contexts. U.S. based researchers must submit an IRB proposal to such a board before the research can begin. The board independently reviews the proposal to make sure that risks to participants are minimized (with harm defined not only as physical, but mental or social).
Participants must also be aware of the nature and purpose of the study, and be freely able to give or withhold their consent to participate.
Human subject review boards are not commonly found in museums, so lack of IRB approval can be a problem for anyone not from a university who wants to perform educational research. Some school districts may have associated IRBs. If you partner with an educational researcher, you not only gain expertise for developing the research project, but you will be able to work with them to get IRB approval through their organization. If there is no one you can partner with at a local university, it may still be possible for you to contact its IRB and have it review your research proposal.
Finally, if you do not have a university nearby that you can go to, there are online companies that provide IRB services for a fee (which can be high, as they cater to medical researchers). Alternatively some institutions, like Wayne State University, do not charge fees for IRB review for researchers from non-profit organizations.
Planetarium education research projects are often given exempt status by the IRB. This means that, after an initial review, the project no longer requires continued oversight or periodic review by an IRB. The exemption can occur because the research is looking at educational strategies or practices, involves the use of test scores, or involves the observation of public behavior. Often exemption is given because the educational intervention is not significantly different than what the students would be experiencing in their classes if the research is not occurring. However, even if you think your project should be exempted, the final decision for exemption still must come from the board.
Part of writing an IRB proposal includes considering how someone will be affected by his or her participation in your research. You should always consider how the research will both maximize possible benefits and minimize risks to participants. You may also need to think about the costs of participation: if someone is taking time out of their day to be involved, they may need to be compensated. The compensation could be monetary, and hence needs to be budgeted. Alternatively, it may be easier for a planetarium to offer free passes to the general public in lieu of a cash payment.
Whichever review boards you decide to use, make sure you understand the timelines that they follow for looking at proposals and granting approvals. A review process can take a month or more, and you may have to re-adjust your research plan and re-apply again. Also an institution may have multiple review boards, so look for the one dealing with social science research rather than medical research, for example.
The IRB process may seem intimidating. The board does not want to prevent you from doing the research, but only to make sure that it is done ethically. The best way to work with the IRB is by providing as much information as possible, and making sure you stay in communication with the board members.
In this section, we will consider several avenues for research dissemination. Engaging with a network of peers, including researchers and practitioners, allows researchers to receive feedback and have their work reviewed, which is important to improve the research output; participation in this network also creates opportunities for developing partnerships and collaborations that further research efforts on planetarium education. It is important to share our research results with other planetarium practitioners, as well as educators in related fields, because research findings should be used to improve our practice and to generate new innovations.
By bringing together both education professionals and researchers, conferences create opportunities for networking and allow researchers to stay abreast of current research as well as innovations in instructional design.
There are conferences addressing almost every area of science education and research. Although planetarium conferences are one avenue for dissemination, there are other types of meetings that could relate to educational research in the planetarium, (see Table 3). As we discussed earlier, your research may fit within one of the four quadrants relevant to planetarium-based research. If your research involves informal learning, you may find presenting your work at the Association for Science and Technology Centers annual conference to be valuable. Similarly, if you research engages formal education environments, you may find receive useful feedback by presenting your work at science teacher conferences, such as the National Science Teacher Association annual conference.
We also encourage you to consider publishing your research, particularly in Planetarian, as this will allow you to reach a wide audience of readers, beyond those who attend planetarium and other conferences. Further, journals are also a more permanent form of dissemination, helping to build the knowledge-based on planetarium education by allowing for more opportunities for others to learn from your research.
As with conferences, journals are aimed towards different audiences, including separate journals for researchers and practitioners in formal and informal environments. Table 4 includes some of our suggestions of potentially relevant journals, though this list is not exhaustive.
The first step to publishing your research is to select the type of journal appropriate for your work. The best way to do this is to look at the journal issues themselves and consider whether the type of articles being published are similar to the nature of your work. While Planetarian and IPS affiliate publications should be considered, there may be other journals appropriate for publishing your work. It is helpful to look at the articles you are citing in your manuscript; if you are not citing papers from the journal you are considering, that journal may not be a good fit or you may need to spend more time connecting your work to those research publications.
Consider the audience you are trying to reach with your findings. Journals are designed to cater to specific readers; some are primarily aimed towards other researchers, while others aim towards practitioners. Once you have found the journal that you would like to submit to, the website of the journal will have the next steps to follow with regards to formatting and how to submit.
The submission process for science educational research journals is similar to other research fields (e.g., astronomy research journals). After submission, the editorial staff will review the manuscript and if it is a good fit, the manuscript will be sent out for review. (All research journals and many journals aimed at practitioners include a peer review process.) The review process can take as little as a few weeks but can often take 3-4 months, depending on the journal. The editor then considers the reviews and makes a decision whether to accept the manuscript as is, accept with further revisions, reject but encourage resubmit, or reject.
If your manuscript is rejected, carefully consider the feedback and use that to revise your manuscript before sending it to another journal. Rejections are a normal part of the process of publishing your work and should not dissuade you from continuing to submit your work in other journals. If your manuscript is rejected but you are encouraged to resubmit, or it is accepted with revisions, pay close attention to the feedback you receive and do your best to revise accordingly. While you do not need to make all the changes that reviewers suggest, you will need to provide a justification for any suggestions that you do not include in your revised manuscript.
Finally, consider the value of local planetarium association newsletters. Most are eager to receive and print submissions from their geographical areas, and many strive for high-level production and service to their readers. These affiliate newsletters also may serve as starting points for publication (or republication, with permission) in other journals.
Engaging in educational research is a time-consuming, yet rewarding process. We hope this summary provides a starting point of ideas and resources for those interested in starting to engage in their own research projects. Developing a research culture in the planetarium field has the potential to greatly benefit our broader community, if we begin to engage in a dialogue around the results of these studies.
Allen, S. (2002). Looking for learning in visitor talk. In Learning Conversations in Museums (pp. 259–303). Lawrence Earlbaum Associates.
Barab, S. (2014). Design-based research: A methodological toolkit for engineering change. Cambridge Handbook of the Learning Sciences, 2nd Edition (R. Keith Sawyer, Ed.), New York, Cambridge University Press, pp. 151-171.
Brazell, B.D. & Espinoza, S. (2009). Meta-analysis of planetarium efficacy research. Astronomy Education Review, 8, 010108-1-11.
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-42.
Chastenay, P. (2015). From geocentrism to allocentrism: Teaching the phases of the Moon in a digital full-dome planetarium. Research in Science Education, 1-35.
DeWitt, J. & Osbourne, J. (2007). Supporting teachers on science-focused school-trips: Towards and integrated framework of theory and practice. International Journal for Science Education, 29(6), 685-710.
Gillette, S. (2014). The effects of seductive details in an inflatable planetarium. Planetarian, 43(2), 26-28, 30.
Griffin, J. (1998). School-museum integrated learning experiences in science. Unpublished doctoral dissertation. University of Technology, Sydney, Australia.
The Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8.
Falk, J. & Dierking, L. (2000). Learning from Museums: Visitor Experiences and the Making of Meaning. Altamira Press: Walnut Creek, CA.
Falk, John H., Dierking, L., & Adams, M. (2006). Living in a learning society: Museums and free-choice learning. In A Companion to Museum Studies (S. Macdonald, Ed.). Oxford: Blackwell, pp. 323–339.
Lelliot, L. D. (2007). Learning about astronomy: A case study exploring how grade 7 and 8 students experience sites of informal learning in South Africa. Unpublished doctoral dissertation. University of Wiwatersrand, Johannesburg, South Africa.
Orion, N., & Hofstein, A. (1994). Factors that influence learning during a scientific field trip in a natural environment. Journal of Research in Science Teaching, 31(10), 1097-1119.
MacDonald, S. (2007). Interconnecting: Museum Visiting and Exhibit Design. CoDesign: International Journal of CoCreation in Design and the Arts, 3(S1), 149-162.
Mayer (2005). Cambridge Handbook of Multimedia Learning. Cambridge: Cambridge University Press.
National Research Council (2002). Scientific Research in Education. (R.J. Shavelson, L. Towne, Eds.). National Academies Press
National Research Council (2007). Taking science to school: Learning and teaching science in grades K-8. (R.A. Duschl, H.A. Schweingruber, & A.W. Shouse, Eds.). Washington, DC: National Academies Press.
National Research Council (2009). Learning Science in Informal Environments: People, Places, and Pursuits. Washington, DC: National Academies Press.
NGSS Lead States. 2013. Next Generation Science Standards: For States, By States. Washington, DC: National Academies Press.
Piaget, J. (1970). Piaget’s theory. In L. Carmichael (Ed.), Carmichael’s Manual of Child Psychology (pp. 103-128). John Wiley and Sons.
Plummer, J.D. (2009). Early Elementary Students’ Development of Astronomy Concepts in the Planetarium. Journal of Research in Science Teaching, 46(2), 192-209.
Schmoll, S. (2013). Toward a Framework for Integrating Planetarium and Classroom Learning. Unpublished doctoral dissertation. University of Michigan.
Slater, S. J., Slater, T. F., & Bailey, J. M. (2010). Discipline-Based Education Research: A Scientist’s Guide. New York: WH Freeman.
Slater, S. J., Slater, T. F., & Shaner, A. (2008). Impact of backwards faded scaffolding in an astronomy course for pre-service elementary teachers based on inquiry. Journal of Geoscience Education, 56(5), 408.
Small, K.J. & Plummer, J.D. (2014a, Summer). Impact on children’s conceptual constructs regarding observational features of the Moon: A look at elements of program and instruction design for early elementary-aged students. Constellation, 2-4, 7.
Small, K.J. & Plummer, J.D. (2014b). A longitudinal study of early elementary students’ understanding of lunar phenomena after planetarium and classroom instruction. Planetarian. 43(4), 18-21.
Sweller, J., Van Merrienboer, J. J., & Paas, F. G. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251-296.
Zimmerman, L., Spillane, S., Reiff, P., & Sumners, C. (2014). Comparison of student learning about space in immersive and computer environments. Journal and Review of Astronomy Education and Outreach, 1(1), A5-A20.
Vygotsky, L.S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.
Yu, K.C., Sahami, K., Sahami, V., & Sessions, L. (2015). Using a digital planetarium for teaching seasons to undergraduates. Journal of Astronomy & Earth Sciences Education, 2(1).
Table 1. Recent research syntheses of science education research
Table 2. Research Methods Resources
Table 3. Conferences for Researchers and Practitioners
Table 4. Journals for Researchers and Practitioners