Main Body
5 Tracking Research Activities and Demonstrating Collective Progress
By Kim Spencer, So Hee Hyun, and Emma Dums
“Collective impact initiatives depend on a diverse group of [individuals] working together, not by requiring that all participants do the same thing, but by encouraging each participant to undertake the specific set of activities at which it excels in a way that supports and is coordinated with the actions of others.” (Kania & Kramer, 2011)
Introduction
One of the main goals of the National Research Mentoring Network (NRMN) Coordination Center was to guide the NRMN research community, including the 11 research teams, to fully realize the national impact and research potential of NRMN Phase II. To achieve this goal, our team prioritized sharing collective progress updates across several main research study components: Institutional Review Board (IRB) protocols, survey measures, study design, study enrollment, survey distribution, data sharing, and dissemination. For each component, we identified key research activities and expectations for the individual research teams and the NRMN Coordination Center.
IRB Protocols
- NRMN Research Team Expectations
- Manage IRB protocol for research study
- Provide participants the option for their data to be used in future research studies
- Share IRB materials with the NRMN Coordination Center
- NRMN Coordination Center Expectations
- Manage IRB protocol for the NRMN Coordination Center, including future data use
- Compile IRB documents across research studies and confirm data sharing language
Survey Measures
- NRMN Research Team Expectations
- Manage IRB protocol for research study
- Provide participants the option for their data to be used in future research studies.
- Share IRB materials with the NRMN Coordination Center
- NRMN Coordination Center Expectations
- Lead discussions and working groups to reach consensus on common measures
- Confirm use of common measures on surveys
Study Design
- NRMN Research Team Expectations
- Determine intervention design
- Share information on intervention modality, duration, and cohort structure
- NRMN Coordination Center Expectations
- Identify any site recruitment overlap across research studies
- Compile information on intervention design components
Study Enrollment
- NRMN Research Team Expectations
- Determine recruitment approach
- Recruit and enroll participants
- Share enrollment numbers with the NRMN Coordination Center
- NRMN Coordination Center Expectations
- Compile information on participant enrollment and outreach
Survey Distribution
- NRMN Research Team Expectations
- Administer surveys
- Manage data collection
- Share survey distribution numbers with the NRMN Coordination Center
- NRMN Coordination Center Expectations
- Compile information on survey distribution
Data Sharing
- NRMN Research Team Expectations
- Manage research study datasets
- Share common measure data and codebooks with the NRMN Coordination Center
- Respond to questions about the common measure data
- NRMN Coordination Center Expectations
- Determine data sharing windows and custom data sharing timelines for each research study
- Review shared data for common measures
- Create common measures datasets
Outcome Dissemination
- NRMN Research Team Expectations
- Disseminate research study findings
- NRMN Coordination Center Expectations
- Compile aggregate information for webinars, presentations, reports, and other dissemination outlets
Given that a significant part of our role was administrative—centered on tracking research activities and coordinating the use of common measures (see Chapters 2 and 4)—it was important to clearly delineate how the research teams and the coordination center would collaborate to demonstrate collective progress. Once we established these roles and expectations, we shared this information across a variety of settings, including meetings, webinars, and on our NRMN Research Community Google Site.
Establishing Community and Building Trust
An important catalyst for coordination is creating an environment that fosters community and builds trust (Rolland et al., 2011). As mentioned in Chapter 3, a strong communication infrastructure is important for promoting transparency, critical for large-scale projects. To build on communication work implemented during the first several months of the grant, we developed a separate communication strategy for tracking research activities with research managers from each research study. This strategy, focused on building connections, included more direct correspondence and touch points that were customized for each research study. This reflected our overall core values of transparency, accessibility, and trust (for more information communicating with research teams, see Chapter 3).
Our team recognized research managers as the research team members most familiar with their study’s IRB protocol, research design, data collection process, data handling, and day-to-day operations. While the principal investigators and co-investigators also hold this information, the research managers were best positioned to respond to inquiries from the coordination center due to their focus on the research study and in-depth involvement across multiple research activities. Research managers across all 11 NRMN research studies brought diverse backgrounds and varying levels of experience. To better understand this range, we periodically administered surveys to gather information about their roles, including the time they dedicated to key study activities such as grant administration, IRB protocols, study recruitment, and data handling (see Appendix 8: Research Manager Scope of Work Survey).
Communication with research managers evolved throughout the grant to accommodate different needs. Early in the grant, we held large group meetings with the research managers from all 11 research studies to provide clarity around expectations, roles, and processes. As the research studies advanced, we held meetings with individual research managers to discuss project nuances, deepen our understanding of their research design, and onboard new study team members. While large team meetings were still important, this purposeful shift allowed for more in-depth sharing and relationship building. Throughout the grant, NRMN Coordination Center team members also sent emails to share and request information from individual research managers. The main communication strategies included:
- Large group research manager meetings: Meetings with all research managers were held in year 1 of the grant to create space for research managers to connect and find community, share information, identify needs across the group, and review expectations for research activity tracking.
- Individual research manager meetings: Meetings with the research managers from each research study were held in the beginning and middle of the grant (years 1-3). These meetings were used to confirm our understanding of each study design, discuss common measure usage, align data sharing expectations, and onboard new research team members.
- Individual research study emails: The NRMN Coordination Center sent customized emails to each research team throughout the grant. Emails included data and tracking updates, requests for study information or data, and questions related to data issues in need of clarification.
To ensure we met the peer support needs of the research managers, we also established a community of practice with a focus on peer support and professional development. More information on this community of practice is described in Chapter 6.
Tracking Research Activities
When setting up the infrastructure to track research study activities, our team was intentional about designing a system that promoted collaboration and accessibility. The tracking infrastructure needed to include features that supported the following:
- Transparency and information sharing
- Collaborative access and accountability
- Flexibility for customization and growth
- Regular updates and clear record keeping
To meet these goals, our team leveraged Box and Google Workspace, both of which were available through the University of Wisconsin–Madison. Box was used to store research study data, and Google Workspace was used for collaborative work such as creating agendas and tracking research activities (more information on the decision to use Google Workspace for coordination center work is described in Chapter 3). We chose University of Wisconsin–Madison Box to store data files instead of Google Workspace to comply with IRB policy and ensure data was protected on a secure platform. Both Box and Google Workspace included a file structure where each research study had their own folder with materials that only members of their research team could access. Our team met with each research study to determine which research team members should have access based on their roles. Access levels were revisited throughout the grant to promote transparency, support ongoing collaboration, and maintain accurate record keeping.
Given the high variety and volume of research activities being tracked for each research study, we decided that a multi-tab Google spreadsheet would work best to keep all information in one place. Each study spreadsheet had separate tabs for study design, intervention timelines, recruitment strategies, participant enrollment, and survey administration. Later, we added new tabs to map the usage of common measures and share information about data transfer plans (see Appendix 9: Example of Research Study Tracking Spreadsheet). We customized the spreadsheets for each research team, allowing for variations in how activities were tracked based on study design. For example, spreadsheet tabs were adjusted to include specific design components and accommodate the different number of intervention groups and survey timepoints. This flexibility enabled our team to capture the unique aspects of each research study while maintaining centralized tracking.
Throughout the grant, both our team and the research managers added information to the study spreadsheets to keep track of progress. As research studies advanced and we learned more about their individual research designs, our team identified information that should be added to their specific spreadsheet, which we noted for upcoming information requests. For example, once we knew when an intervention would take place, we made a note to ask the study’s research manager to add the number of participants enrolled and the number who completed the baseline survey. Our team sent information requests several times during the grant year, intentionally grouping requests together to limit the number of emails sent to each study team.
To record information requests, our team created an action item spreadsheet to organize the inquiries sent to each research study (see Appendix 10: Example of Action Item Spreadsheet). In the spreadsheet, each research study had a separate tab and requests were categorized by time period and the type of research activity. Tracking all of the requests in one spreadsheet promoted synergy within our team and helped organize information. We used columns to note important dates, including the initial request, reminders, and when the information was received. While this level of record keeping required a substantial amount of time, it provided important documentation that helped prevent duplicate efforts among our NRMN Coordination Center team and revealed communication issues or patterns across research teams. Overall, research managers responded well to this structured system. In cases where research managers were unresponsive to information requests, we included additional research team members on email correspondence, such as principal investigators, to ensure that requests were received.
Our team supported this work through partial appointments, balancing our commitment alongside other university grants and projects. The team included a co-investigator who oversaw tracking coordination; a researcher who led infrastructure management, study tracking, and correspondence with research managers; a researcher who led data management and cleaning; and several additional team members who provided tracking and dissemination support.
Tracking Common Measure Usage and Requesting Common Data
Tracking Common Measure Usage
From the beginning of the grant, our team emphasized the importance of all research studies using common measures to assess key constructs and hallmarks (for more information on using common measures, see Chapter 4). As such, it was important for our team to develop an initial process plan to track which common measures each research study used in their surveys to research study participants.
Tracking common measure usage supported several main objectives:
- Hold research teams accountable for using agreed upon common measures.
- Provide the coordination center with a detailed understanding of each research study’s survey design.
- Inform data sharing plans customized to each research study.
The first step of the process included reviewing all research study surveys and identifying which common measures were included (Figure 6). In some instances, surveys were missing a measure that was required (e.g., gender). For these cases, we followed up with the research managers to ensure the measure was added. At this stage, we were not assessing the accuracy or consistency of their usage, only inclusion (for more information on measure review, see Chapter 7). Once measure usage was confirmed, we used this information to map which measures would be asked across study timepoints. We added the information to the research study spreadsheet and shared the mapping results with the research managers to confirm accuracy (see Appendix 11: Example of Measure Mapping Spreadsheet). To demonstrate the collective progress of this work, our team compiled information on common measure usage and shared it with the NRMN research community across a variety of platforms (see Chapter 4 Figure 5).
Figure 6. Initial process for mapping common measure usage
Requesting and Managing Common Measure Data
Early on, we identified the creation of a multi-study dataset as one of the main outputs of our grant. For this dataset to be open for use in future research, the coordination center managed an IRB protocol at the University of Wisconsin–Madison to allow the secure transfer of data. Since each research study collected data continuously throughout the grant, we established data sharing intervals to ensure data was shared periodically. This approach created opportunities to collaborate with research managers in addressing data concerns and allowed us to build the common measures datasets as data became available rather than waiting until the end of the grant.
As noted in Chapter 4, we worked with each research team to incorporate data sharing language into their IRB Protocol and informed consent materials (see Appendix 6: Template for IRB Protocol Language and Appendix 7: Example of Internal Data Sharing Plan). This ensured that research studies could share data with the NRMN Coordination Center. To facilitate regular data sharing, we created separate data sharing plans for each research study that reflected their unique data collection design and IRB protocol (see Appendix 12: Example of Customized Research Study Data Sharing Plan). Following their data sharing plan, we sent data requests to research managers via email at regular intervals, taking into account data collection timelines and our team’s capacity. We instructed research managers to upload their survey data of common measures into their secure University of Wisconsin–Madison Box folder. Once uploaded, we reviewed each data file and deleted any non-common measure data that was included. Our team used this data to build a common measures dataset for each research study. See Chapter 7 for more information on the data request and management process, including the steps taken to clean data and resolve data issues.
We sent the first round of data requests in spring of 2021, nearly two years after the start of the grant. This timing was intentional as it allowed us to complete several key milestones, including finalizing the list of common measures, integrating and approving IRB data sharing language, establishing key infrastructure (i.e., Google Workspace and Box), resolving any early data sharing concerns, and setting expectations for the data sharing process. Additionally, this timeline allowed for research studies to enroll participants, collect data, and prepare datasets for sharing with the coordination center. Perhaps most importantly, this time provided the opportunity for the coordination center to establish trust with the research teams.
Throughout the grant period, we sent a total of 63 data requests and at least 70 additional follow-up emails with reminders or clarification about the data. The final data requests were sent two months before the official end of the five-year grant period, giving research managers sufficient time to respond and allowing our team to follow up with any questions or clarifications. This systematic and structured approach to requesting and managing data ensured that data was shared securely and consistently, ultimately contributing to the development of a robust, multi-study dataset that became a central element of the NRMN Coordination Center’s work. It is worth noting that several teams received no-cost extensions to continue data collection beyond the grant period. We did not request their additional data because the data would be out of scope of our grant obligations and would have required additional resources and personnel that we did not have.
Showcasing the Collective Progress
One of the most rewarding outcomes of coordinating research activities is being able to demonstrate collective progress. Throughout the grant, we compiled information on research study progress and created visuals to share with the research community. In addition to text, incorporating visuals helped us make complex information more accessible, enhanced communication of findings to wider audiences, and increased engagement within the research community. The regular dissemination of collective progress supported several main objectives:
- Hold research studies accountable for sharing information on their research activities.
- Acknowledge the progress and important work being accomplished across NRMN Phase II.
- Build trust with how research study data is being used and showcased.
- Emphasize the interconnectedness of the research community’s work.
- Create opportunities to celebrate milestones
- Inspire new ideas for individual or collaborative work.
Although we organized information on research activities for each research study, we reported aggregate information, intentionally avoiding identification of specific research studies. This approach ensured that no research study was singled out, which was especially important since each research study had a unique study design. For example, one research study had a goal of enrolling fewer than 300 participants, whereas others aimed to enroll over 3,000. By reporting participant enrollment numbers across all research studies, we placed the focus on the collective impact of the research community. We chose to display information by quarter so that information was aligned with the grant reporting cycle. Further, quarterly reporting provided opportunities to identify more meaningful trends across the research studies and do so in a way that was not burdensome for us to compile.
One of the most impressive research activities that we frequently highlighted in webinars was the volume of surveys distributed. As with participant enrollment, we compiled the number of surveys that were distributed to participants each quarter (Figure 7a) We also included the cumulative number of surveys distributed to date (Figure 7b). For both figures, we decided to include all surveys that were sent to participants, including baseline surveys, immediate post-surveys, 6-month post surveys, and other longitudinal surveys. These collective metrics were a concrete and accessible way to convey the research community’s momentum and underscore the meaningful progress achieved through our collective effort.
Figure 7a. Participant surveys distributed by quarter
Figure 7b. Cumulative participant surveys distributedConclusion
Coordination centers have the exciting but often complex and challenging role of demonstrating collective progress and impact. Creating an environment that fosters community and builds trust is essential for strong communication and information sharing. It is critical to identify roles early on and set expectations for how research teams will work together towards a common goal. This, along with robust tracking systems, will strengthen communication, promote transparency and collaboration, and enhance accountability. Dedicating time to determine which information best demonstrates collective progress will help prioritize how that information is tracked and used. When possible, share regular progress updates, create visuals to make information more accessible, increase engagement with the research community, and inspire collaboration.
Lessons Learned
- Prioritize building relationships with research managers. Research managers are invaluable members of research teams and building relationships with them should be prioritized.
- Provide training opportunities to research team members. Research team members have varying levels of experience and comfort working with data. Additional training may be needed to ensure clear expectations and requirements.
- Acknowledge needs and provide resources. Acknowledging research team member needs, making adjustments to accommodate work styles, and creating supportive resources will build trust with the research community.
- Provide onboarding to align expectations. Routine onboarding with new research team members is necessary to ensure clear expectations.
- Be proactive about checking in with research managers. Research studies may evolve throughout the grant, so it is important to proactively check in and inquire and stay informed about changes to study.
- Create tracking materials to showcase project outcomes. Take time to consider the types of information that will showcase project outcomes and use backward design to create tracking materials that allow data to be easily compiled.
- Consider what information is needed for reporting. Carefully consider what information will be needed for creating reports and presentations that will be shared with collaborators (e.g., funders, program officers, steering committee members, institutional leaders) and what will be useful for demonstrating collective progress with community members. Some data, such as information on the career stage of participants, may not be publicly available or found in data shared with your coordination center. Create plans to obtain this information if needed.
Acknowledgements
We acknowledge Dr. Jenna Rogers for her work in establishing processes for coordinating research activities and Mr. Jimmy Robinson for his support in tracking and compiling research study information.
