A Comparative Analysis of Information Gathering Meetings Conducted by Novice Design Teams Across Multiple Design Project Stages Free

Information gathering activities are core parts of engineering design processes. Designers rarely begin design work with sufficient knowledge about their problem context to develop successful solutions [1,2]. As such, information gathering activities play an important role in enabling designers to define their design problems and evaluate solution feasibility [3–5]. There are many methods that designers may adopt to gather information, including meeting with project stakeholders and domain experts, benchmarking existing products, searching academic research and patent databases, and researching relevant engineering standards [2–8].

Meetings with stakeholders and domain experts—including, but not limited to, informational interviews, project planning meetings, and collaborative design sessions or workshops—can have a unique impact on engineering design outcomes compared to other types of information gathering methods. For example, previous studies have shown that stakeholders can provide specific, in-depth information about their needs and lived experiences that may challenge the preconceived notions of designers [8–11]. Recommended practices for conducting effective information gathering meetings include soliciting diverse perspectives [8,9,12,13], using appropriate information gathering techniques during meetings [7,14–16], and conducting multiple information gathering meetings with key stakeholders over time [17–19]. These practices can help designers elicit useful and comprehensive information through their meetings.

The development of engineering design expertise related to conducting information gathering meetings has been relatively underexplored in the literature. Few comprehensive studies of novice engineering designers’ approaches exist that simultaneously explore the types of individuals from whom novice engineering designers gather information, the techniques employed by novice engineering designers to solicit information, and the quantity and timing of their meetings. As a result, there may be specific aspects of novice engineering designers’ approaches to conducting information gathering meetings that diverge from recommended practices and that prior work has not described in depth. To address this research gap, our study analyzed data related to information gathering meetings conducted by novice mechanical engineering designers in their culminating design experience prior to becoming engineering practitioners.

The phrase “information gathering meetings” represents an inclusive way to describe a range of conversation-based interactions between designers and stakeholders or domain experts that designers use to gather design-relevant information. The literature we reviewed includes those related to interviewing best practices (e.g., [20,21]) and interviewing for empathy (e.g., [15,22]), as well as stakeholder engagement (e.g., [10,23]) and design ethnography (e.g., [8,24]) more broadly.

Previous studies and textbooks in the fields of engineering design, design ethnography, requirements engineering, and human–computer interaction (HCI) have provided several recommendations to guide engineering designers in conducting effective information gathering meetings. We synthesized these recommendations into three distinct recommended practices: (1) solicit diverse perspectives, (2) use appropriate information gathering techniques, and (3) conduct multiple meetings over time. Our subsequent discussion of background literature includes recommendations related to each practice and highlights some of the ways that these practices may influence one another.

Recommended practices include that designers should conduct information gathering meetings with multiple individuals possessing various knowledge and representing diverse viewpoints. Engineering design textbooks [3–5] suggest that designers should meet with both stakeholders (i.e., individuals that will be directly or indirectly affected by project outcomes) and domain experts (i.e., researchers or practitioners who possess project-relevant engineering expertise) to inform their design decisions. Rosenthal and Capper [8], in their review of ethnographic methods used across 14 different product design contexts, echoed this point by highlighting the range of different consumers and professionals that designers may consult to inform their design decisions. Studies further show that differences in stakeholder or domain expert characteristics, for instance related to age and physical ability [9,22], personal or cultural values [8,25–27], social status within a larger community [13,26–29], and professional background [12,30,31], can lead different individuals to provide significantly different and sometimes contrasting descriptions of experiences or feedback on design concepts. By sampling diverse perspectives, designers may be able to identify and account for a wider range of stakeholder needs and requirements while designing. However, as emphasized by Ogbonnaya-Ogburu et al. [13] in their HCI work, designers must also reflect on how their personal identities (e.g., race, gender, and socioeconomic status) may influence the willingness of stakeholders with dissimilar identities to share information during meetings. Designers that lack this self-awareness risk perpetuating existing societal inequalities with their design solutions, even in cases where they solicit diverse stakeholder feedback.

Furthermore, designers should employ information gathering techniques during meetings that are appropriate for the information that they hope to gather. Common techniques include exploring an individual’s knowledge or experiences using a structured interview protocol [7,15,20,21], inviting stakeholder participation in design activities to leverage stakeholders’ unique knowledge and experiences [7,10,23,27,29], and developing a mutually accessible design language with stakeholders to facilitate problem co-exploration [11,14,32]. Designers may also leverage design representations such as prototypes during information gathering meetings, for instance to facilitate communication with stakeholders or domain experts [11,33,34] or explore stakeholders’ preferences [19,22,34].

Different information gathering techniques differently affect the types of information that stakeholders and domain experts may be able or willing to share during information gathering meetings. For example, while semi-structured exploratory interviews can obtain deep insights about specific experiences, they are not always effective tools for eliciting knowledge that is tacit or otherwise difficult to articulate [2,7,24]. Mazzurco et al. [16], based on their review of community engagement methods in design for development contexts, suggested that highly participatory (or “coconstructive”) information gathering techniques enable stakeholders to present their perspectives in authentic ways that can reveal otherwise inexpressible knowledge. Aguirre et al. [35], in their study of co-creative service design events, have also noted that different participatory information gathering techniques encourage different types of thinking (e.g., abductive thinking versus perspective-taking) and thus elicit different types of stakeholder or domain expert responses.

The appropriateness of certain information gathering techniques also depends upon the characteristics of the stakeholders or domain experts from whom designers are gathering information. For instance, in a study of engineering design interviewing, Deininger et al. [12] demonstrated that stakeholders with different levels of relevant domain knowledge may differ in their ability to provide in-depth and useful responses during exploratory interviews. Designers may thus need to adapt their information gathering techniques to best suit the individuals from whom they are gathering information.

Recommendations also include conducting multiple information gathering meetings with key stakeholders throughout design projects. Studies suggest that soliciting stakeholder perspectives throughout problem definition [29,36] and concept development processes [18,19,22,37] can help designers develop more appropriate solutions. For example, Tiong et al. [19] investigated the use of prototypes in the development of three different financial technology business-to-consumer products to identify economically optimal prototyping approaches. Their findings emphasized that designers should create low-fidelity prototypes early in their design processes to test their assumptions and uncover users’ mental models and then utilize higher fidelity prototypes during later design stages to gather more detailed information. In addition, conducting multiple meetings with the stakeholders can facilitate rapport-building and lead stakeholders to provide more open responses in later meetings [20,28,38]. Agid and Chen [17], in their study of community-engaged, collaborative design projects, and Pahl and Beitz [4], in their textbook on engineering design, also emphasized that stakeholder perspectives on needs and requirements often change over time. As a result, designers may need to meet with stakeholders several times to gain comprehensive information.

In some cases, designers may influence changes in stakeholder perspectives through their choices of information gathering techniques. For instance, previous studies have documented how participatory information gathering techniques, when used by designers across multiple meetings, may lead stakeholders to develop new design knowledge that in turn influences their perspectives of the design problem [17,23,39]. In such situations, stakeholders may provide new and unexpected information. As an example, Taffe [39] examined how end-users in a service design context responded to a series of three co-design workshops and, contrary to expectations, found that workshop participants began imagining the needs of other end-users rather than discussing their own experiences. Designers should thus consider how their information gathering techniques in earlier meetings may impact stakeholder or domain expert responses in later meetings.

Studies of novice design approaches provide an informative lens for understanding the evolution of design expertise. For example, studies such as Luck [40] and Deininger et al. [41] analyzed novice design approaches to better understand the baseline knowledge or skills that designers may possess related to gathering information. Luck [40] found that novice architects struggled to adopt stakeholder language when gathering information, while Deininger et al. [41] found that novice engineering designers struggled to leverage tools such as prototypes effectively to gather information. Other studies by Loweth et al. [42] and Bano et al. [43] documented additional challenges encountered by novice engineering designers while gathering information, such as difficulties formulating open-ended questions and exploring stakeholder or domain expert experiences in depth.

Previous studies also observed significant variation among novice engineering designers’ approaches to gathering information. For example, novice engineering designers have been shown to vary in the amount of information that they gather to define their design problems [36,44] as well as the timing and quantity of information gathering meetings that they conduct over the course of their design projects [18,45]. Notably, differences in design context do not seem to explain these observed variations, since participants in the aforementioned studies were working on the same or very similar design tasks. Zoltowski et al. [46] and Loweth et al. [45] also showed that novice engineering designers possess varying perspectives regarding the value of stakeholder input to inform their projects. These perspectives may influence when and how novice engineering designers conduct information gathering meetings.

However, previous findings provide only partial insights regarding novice engineering designers’ information gathering approaches since few studies have described the content of novice designers’ information gathering meetings in depth. Studies such as those by Luck [40] and Loweth et al. [42] that identified specific techniques used by novice architects and engineering designers to gather information provided no indication of how often novice designers employed these techniques. Previous studies that explored other aspects of novice information gathering approaches, such as the quantity and timing of conducted meetings (e.g., [18]) or the different types of individuals from whom novice engineering designers gathered information (e.g., [47,48]), did not analyze data on the content of novice engineering designers’ meetings that might explain differences in approaches or experiences observed across participants. Deep exploration of novice engineering designers’ information gathering meetings is needed to further understandings of novice information gathering approaches and to reveal additional knowledge gaps that training, tools, and pedagogy can be developed to address.

The goal of our study was to understand how novice engineering design teams conducted information gathering meetings to inform their design projects. Our study was guided by the following research questions:

1. With whom do novice engineering design teams meet to gather information?

2. When do novice engineering design teams conduct information gathering meetings?

3. How do novice engineering design teams solicit information during their information gathering meetings?

Participants included 24 students from six design teams enrolled in a single-semester senior-level mechanical engineering capstone design course at a large Midwestern university. This capstone course represented a culminating educational experience that was meant to prepare students to enter professional engineering practice. The design behaviors that participants exhibited in this capstone context are thus likely indicative of how they would approach their early-career design work. Our sample size was appropriate for the in-depth research methods leveraged and was larger than comparable studies (e.g., [47,49]) of design team communication and information gathering that investigated one to three teams.

Each design team was composed of three to five undergraduate students who were majoring in mechanical engineering and who had completed the required two-course mechanical design sequence. Many participants also described additional design experiences in curricular, co-curricular, and internship settings. However, only one participant reported prior experiences conducting information gathering meetings to inform design projects, since this material was not a focus in participants’ earlier mechanical design courses. The composition of the six participating teams and their project foci are included in Table 1. Table 1 also includes information about the number of information gathering meetings conducted by each team; this information is discussed in greater depth in Sec. 4.2.

Each team was tasked with developing a prototype to solve a unique design problem experienced by a local sponsoring group or organization. Teams worked on one of three types of projects (all project descriptions have been anonymized to protect student and stakeholder identities):

Developing assistive device: Teams A, B, and C were developing assistive devices to help young adults with physical disabilities complete day-to-day tasks, with each team working on a separate project. Each team received an initial project description such as the following: “[Task] seems like a simple mechanical task, but in fact it requires very complex motion and sensory information processing. This project is focused on creating a new, low-cost device which can be employed by wheelchair users and can adapt to [a variety of situations]. The device will be electrically/pneumatic powered and will require minimal tele-operation…” In addition to a project sponsor, each team was also assigned a specific user who would use their device. All three project topics had been partially addressed in a previous semester by other capstone teams, but none of the solutions produced in this previous semester fully met the users’ needs. As such, significant design iterations were still necessary for all three projects, and Team C in particular chose to generate and develop completely new solution concepts.

Modifying university space: Teams D and F were proposing solutions to address chronic issues experienced by different buildings associated with the university. Initial project descriptions of this type resembled the following: “[The building location] is owned and operated by [university organization], and is located in [Midwestern town]. The [building] is used by [a variety of people], and [the issue becomes most severe] during times of heavy use. The goal of this project would be to make recommendations for improvements to the space [to address the issue], and possibly demonstrate a proof-of-concept prototype(s) …” As part of their projects, Teams D and F were both tasked to create detailed plans describing how their project sponsors should upgrade their respective buildings using pre-existing materials or technologies. They were also encouraged to develop physical and/or virtual prototypes to demonstrate the efficacy of their improvement plans.

Developing measurement tool: Team E was developing a measurement tool to evaluate the safety of a common consumer product. Their initial project description was as follows: “Design a tool to test [the displacement] made when [the user is contacting the consumer product]. Ideally, the tool should be easily transportable so that investigators can employ it at investigation sites. In the past, researchers have tried to create a measurement device … [However,] this [previous] tool is limited because the measurement is not precise, it is not a good approximation of what happens when [the user contacts the consumer product], and it has low reliability.”

As part of the capstone course, team projects proceeded through five main design phases. Teams submitted a design report at the end of each phase. Design Phase 1 focused on problem definition and requirements development. Design Phase 2 focused on concept generation and selection. Design Phase 3 focused on engineering analyses and development of solution prototypes. Design Phase 4 focused on iterative design and testing of prototypes. Design Phase 5 focused on verification and validation of prototypes, and teams submitted final reports summarizing their work across the semester at the end of this last phase. The relative duration and sequencing of these design phases is shown in Fig. 1. Although each team received an initial project description at the beginning of the course, this project description did not provide all the information needed to develop an effective solution. As such, the capstone context provided an opportunity to study how novice designers gathered additional information related to their design projects through in-person, phone, and video meetings across multiple design stages.

Teams attended two lectures during Design Phase 1 related to gathering and utilizing stakeholder and contextual data to inform their design projects. The first lecture described recommended practices for conducting design interviews (primarily drawn from Refs. [47,50]) and provided suggestions for identifying project stakeholders (primarily drawn from Ref. [51]). The second lecture discussed recommended practices for synthesizing stakeholder data into design deliverables such as requirements and specifications (primarily drawn from Refs. [3,52]). In addition to these two lectures, teams were strongly encouraged to consult auxiliary resources related to conducting design interviews that were available through the University of Michigan’s Center for Socially Engaged Design [53]. The capstone course was also divided into multiple sections, two of which were represented in this study. Each section was led by a different mechanical engineering instructor who provided design feedback to teams throughout the semester during weekly project meetings and in response to submitted design reports and student presentations.

Although material related to gathering and utilizing stakeholder and contextual data was primarily covered during Design Phase 1, teams were strongly encouraged to meet with stakeholders and domain experts throughout their projects, for example to validate their user requirements, to solicit feedback on their proposed solutions, and to refine details of their solutions. Aside from a course-required initial sponsor meeting, teams arranged stakeholder or domain expert meetings throughout the semester at their own discretion and/or in response to feedback from their capstone section instructor.

We collected several types of data from participants, including (1) semi-structured researcher interviews with participants, (2) recordings of information gathering meetings conducted by participants, and (3) a list of information gathering meetings conducted by each team. A timeline of these data collection activities is included in Fig. 1.

Through three rounds of semi-structured researcher interviews, we collected data about the individuals from whom each team gathered information, when these meetings occurred, the approaches that teams adopted in preparing for and conducting their meetings, and the ways that teams used the information gathered from their meetings. The first interview occurred at the beginning of the semester and was completed with each participant individually. The second interview occurred after teams conducted their first information gathering meetings and was completed in a group setting with all participating members of the team. The third interview occurred at the end of the semester once teams had mostly finished conducting information gathering meetings and was again completed in a group setting with all participating members of the team. Interview protocols were developed for each researcher interview to ensure comparability across responses [54,55], and each protocol was piloted with undergraduate students who had relevant prior experiences to help iterate on interview questions in advance. In total, our study collected 20 h of audio data from researcher interviews with participants.

In addition, across teams, participants recorded 20 out of the 36 information gathering meetings that they reported conducting as part of their capstone projects. These meetings represented 14 total hours of audio data. A breakdown of the total number of reported meetings and submitted recordings per team is included in Table 1. The average recording length was 41.7 min, with a median recording length of 35.1 min. The interquartile range across meeting lengths (which we used to define the length of a “typical” interview in our analysis) was 27.9–58.3 min. The shortest recording submitted by participants was 4.2 min, while the longest was 83.9 min. Each team also submitted notes and agendas from their meetings, as well as a list containing the dates and individuals present at all meetings that the team conducted (both in person and over audio or video calls) to inform their projects. Team lists and meeting agendas enabled us to confirm participant interview responses regarding the timing of their meetings and the individuals present at their meetings.

The first step in our data analysis was to categorize the types of individuals from whom teams gathered information (addressing RQ1), based on participant descriptions of these individuals in researcher interviews. Teams described gathering information from a variety of individuals, and our goal was to develop categories of individuals that applied across teams. For example, several teams described gathering design feedback from individuals such as sponsors or other stakeholders who were closely related to their design projects. Since several of these individuals seemed to participate in design decision-making, we collectively categorized these individuals as “project partners.” Once our full list of categories was defined, we then reviewed our data again and classified each of the individuals that teams mentioned in their interviews and meeting lists according to one of the identified categories.

Next, we generated timelines mapping out each team’s information gathering process (addressing RQ2). These timelines were based on the list of information gathering meetings submitted by each team and elaborated upon using meeting descriptions from researcher interviews.

Finally, to characterize how teams solicited information during their meetings (addressing RQ3), two researchers coded the 20 meeting transcripts using the list of information gathering behaviors described by Loweth et al. [42]. This list contained 22 behaviors, half of which were more similar to recommended practices for soliciting information during information gathering meetings and half of which were less similar. Behaviors that were more and less similar to recommended practices were also sorted by Loweth et al. [42] into 11 pairings such as encourage deep thinking—elicit shallow responses; delve into stakeholder or domain expert experiences—conflate student and stakeholder or domain expert experiences; and use a co-creative meeting strategy—use a student-centered meeting strategy. These pairings facilitated our ability to discern between different information gathering behaviors when coding our data. For example, we coded instances of participants asking open-ended questions to explore specific stakeholder or domain expert experiences or knowledge as delve into stakeholder or domain expert experiences. Instances where participants mentioned that their own experiences likely resembled the stakeholder or domain expert’s and thus did not ask further questions were coded as conflate student and stakeholder or domain expert experiences. Similarly, we coded instances of participants asking open-ended questions that encouraged stakeholders or domain experts to think critically about a topic as encourage deep thinking. Closed-ended questions that seemed to limit stakeholder or domain expert responses were coded as elicit shallow responses. An example of our coding approach is shown in Table 2.

Our initial round of coding produced a preliminary total for how often each information gathering behavior occurred in each of the submitted meeting recordings. To normalize and compare the frequency with which behaviors occurred across teams, we then developed a four-point ordinal scale to represent behavioral frequencies within each meeting. According to our scale, “none” represented no occurrences of a behavior within the meeting, “present” represented one to two occurrences, “repeated” represented three to six occurrences, and “frequent” represented seven or more occurrences. We defined the four ordinal categories in our scale based upon the size of each team (three to five members) and the typical length of meeting recordings across teams (roughly 30–60 min). After converting our initial occurrence counts for each information gathering behavior to ordinal rankings, we discussed the remaining discrepancies between the two coders’ interpretations and refined our rankings for each behavior across the 20 submitted meetings. The two coders then re-coded five meeting transcripts, specifying the frequency of each behavior within a given transcript according to the ordinal scale. Our inter-rater reliability for this re-coding, calculated using Cohen’s weighted kappa [56] to penalize discrepancies greater than a single ordinal step, was 76.7%. This value represents a reasonably high agreement between the two coders [57], especially since the purpose of our ordinal scale was to facilitate comparison across meeting transcripts rather than define an absolute metric of measurement.

Teams described meeting with four main types of individuals to inform their projects. These types of individuals are summarized in Table 3.

Five out of six teams described meeting with representatives of the group or organization that was sponsoring their design project and/or other closely related stakeholders, i.e., “project partners.” These individuals were either matched to teams through the capstone course as specific “project sponsors” or were invited by project sponsors to participate in team projects. Project partners typically had non-engineering backgrounds and possessed substantial knowledge about stakeholder needs. All six teams met with “domain experts,” or individuals with substantial engineering or design knowledge relevant to the project. These domain experts were usually university professors, although Teams D and E also met with expert consultants. Unlike other teams in our study, Team B’s project was initiated by an engineering professor; as such, Team B did not meet with a separate project partner. Teams A, B, and C were working on project topics that had been addressed in a previous semester. All three teams met with individuals from the previous design team (i.e., the “previous team”) who were able to discuss previous design decisions and could provide suggestions for future design iterations. Finally, Teams A, B, and C were provided specific “users” as part of their capstone projects. Teams B and C interacted with their user, but not Team A. Teams who were not given a specific user to contact also did not meet with users.

Teams reported conducting a total of 36 information gathering meetings throughout the semester, with each team reporting between four and eight meetings as shown in Fig. 2. Many of these meetings (13 out of 36) were conducted during Design Phase 1 while teams were learning about their stakeholders’ needs and developing user requirements and engineering specifications. Each team was required by the capstone course to complete at least one information gathering meeting during Design Phase 1, and all six teams ultimately conducted at least two meetings during this phase.

As teams transitioned to developing and selecting solution concepts (Design Phase 2), differences began to emerge in their respective approaches to conducting information gathering meetings. Teams reported seven total meetings during this phase. Teams D and E met with domain experts to gather more information about what types of solutions might be feasible. Similarly, Team B met with their user for the first time to learn more about their user’s needs. Teams B, C, and D solicited feedback via an information gathering meeting to help them select a solution concept. Teams A and F did not report conducting any additional information gathering meetings prior to selecting a solution concept to implement.

Teams reported five total meetings during Design Phase 3. Teams A and E updated project partners about their progress and discussed the solution concept that they had chosen. This phase was also the last time that Team A reported conducting an information gathering meeting for their project. Teams D and F each met with new individuals to gather additional information relevant to their solution development processes. Teams B and C did not report conducting information gathering meetings during this phase as they completed their engineering analyses and developed detailed solution mock-ups.

Teams reported conducting only two information gathering meetings during Design Phase 4 as they developed and iterated on their physical prototypes. Team E met with project partners to demonstrate an early version of their prototype and solicit feedback. Team C met with their user and project partners prior to beginning their manufacturing process to gather additional information about their user’s physical constraints and solicit further design feedback.

Teams reported conducting nine meetings during Design Phase 5. Team C met with their user and project partners to gather additional contextual information and solicit feedback. Teams C and D also met with domain experts to gather information that could help them implement their final prototype. Lastly, all teams except Team A reported presenting a final iteration of their physical prototype to users and/or project partners prior to submitting their final project report, with Team C conducting two such meetings. Team A did not report a meeting during this phase because they had presented their final concept during Design Phase 3.

Each meeting recording submitted by participants contained a different set of information gathering behaviors. The information gathering behaviors observed within submitted meetings tended to vary based upon the design phase of the meeting, the individuals present at the meeting, and the team who conducted the meeting. A summary of the main meeting contexts where each information gathering behavior was observed to occur and the number of times each behavior occurred per meeting is shown in Tables 4 (for behaviors that were more similar to recommended practices) and 5 (for behaviors that were less similar to recommended practices). The phrase “occurred sparingly” indicates that a given behavior was not typically observed outside of the explicitly noted meeting context; for example, teams did not typically exhibit the encourage deep thinking behavior except in meetings with domain experts. Furthermore, the trends described in Tables 4 and 5 are not mutually exclusive. For instance, Team A typically exhibited the lead the stakeholder or domain expert to conclusion behavior one to two times per submitted meeting during Design Phases 1 and 2. They also, along with the other teams in this study, typically exhibited the verify the conclusions drawn from meetings behavior one to two times per submitted meeting during Design Phases 1 and 2. Team A’s meetings during Design Phases 1 and 2 thus typically contained both behaviors, as well as the other information gathering behaviors that we have described as occurring across all teams or occurring in Team A’s meetings specifically.

Three behaviors occurred most frequently in meetings that teams conducted during Design Phases 1 (problem definition) and 2 (concept development and selection) of their projects: avoid misinterpretations (three to six times/meeting), flexibly & opportunistically probe responses (one to two times/meeting) and verify conclusions drawn from meetings (one to two times/meeting). Teams exhibited the avoid misinterpretations behavior when they asked clarifying questions (e.g., Team C, Phase 2 user and partner meeting: “Do you always wear the seatbelt when you're in the wheelchair?”) or repeated previous responses (e.g., Team D, Phase 1 partner and expert meeting: “And then you also mentioned if our prototype was something you put on the wall or ceilings, you mentioned keeping it to the ceiling for the aesthetics.”). Teams exhibited the flexibly & opportunistically probe responses behavior when they asked additional questions to follow up on information provided by the stakeholder or domain expert (e.g., Team D, Phase 1 expert meeting: “This is kind of a random question… is there cleaning that is associated with [that solution option]?”) Lastly, teams exhibited the verify conclusions drawn from meetings behavior when they checked their understanding of stakeholder or domain expert responses (e.g., Team E, phase 1 partner meeting: “When you say “quantitative,” it seems like that displacement is probably the … most important measurement. Would that be correct to how much the [user] is imprinting into the [consumer product]? Is that the most important metric?”). All three behaviors occurred less frequently in later design phases; participants exhibited the avoid misinterpretations behavior only one to two times per meeting after Design Phase 2 and typically did not exhibit the behaviors flexibly & opportunistically probe responses and verify conclusions drawn from meetings.

Several behaviors occurred mainly with specific types of individuals. The behaviors build rapport with the stakeholder or domain expert (three to six times/meeting), use a co-creative meeting strategy (three to six times/meeting), and assume stakeholder’s or domain expert’s understanding (one to two times/meeting) occurred most frequently in meetings with project partners. The following excerpt from a Design Phase 4 meeting conducted by Team E demonstrates how participants used the behaviors build rapport with the stakeholder or domain expert and use a co-creative meeting strategy to gather information from project partners. In this case, Team E was seeking feedback on a physical prototype of their safety measurement tool from their partners (specifically, their sponsor and a doctor who was also involved with the project):

• Doctor: And then for the weight to mimic [the user]. What are you guys thinking about for how to do that?

• Team E: You have a couple of options there. We were offered brass in the machine shop. But we were also thinking of using steel pellets, which would be smaller so you can add them and increment the weight slower, or by a smaller factor. We're kind of weighing those two options right now I guess.

• Doctor: Oh I get it, you're weighing those two options.

• Team E: Clever, you caught it.

• Sponsor: So you add the weight once [the device] is on the [consumer product]? It's not just a weighted item that you put on it and it goes to wherever it's gonna go?

• Team E: Yeah I suppose that's what we're considering. Whether it would be beneficial just to have three to five standard weight options that you can put in. Or if there would be any benefit to getting a reading more continuously… so you could see how it varies with weight a little bit more smoothly. Those are just things we're considering right now.

• Sponsor: Yeah, the problem I see with pouring in loose pellets or shot or whatever you're using, is the … not for you guys, but for future iterations of the project, someone having to keep track of pieces.

This excerpt aligns with the use a co-creative meeting strategy behavior in two ways. Team E provided space for their project partners to ask questions related to the function of their prototype. Team E’s description of their thought process while considering different weight options also enabled their project partners to provide targeted feedback related to the feasibility of those weight options. Team E’s “weighing” joke (i.e., build rapport with the stakeholder or domain expert) additionally helped them build their relationship with their partner through this exchange.

By comparison, the delve into stakeholder or domain expert experiences behavior typically occurred three to six times in meetings involving either domain experts or users. For instance, participants exhibited this behavior when they asked open-ended, exploratory questions related to an expert’s domain knowledge (e.g., Team E, Phase 2 expert meeting: “If there were to be a time when a [measurement] standard was put into there, what would go into that process?”) or user’s past experiences (e.g., Team B, Phase 2 user meeting: “What were the biggest challenges that you thought [about] using the prototype last semester?”). The behaviors encourage deep thinking (one to two times/meeting) and cede guidance of meeting (three to six times/meeting) also occurred most frequently in meetings with domain experts. The following excerpt from a Design Phase 5 meeting conducted by Team D demonstrates how participants used the encourage deep thinking behavior to gather information from domain experts. In this case, Team D was meeting with an engineering professor from the university to discuss how they might best implement their solution:

• Team D: So we were suggesting … 'Cause the [analysis] that we did showed that the ceiling is the most effective … Like the angled part of the ceiling. And we suggested covering the whole angled part of the ceiling. But [our sponsor] would like to reduce cost.

• Professor: Correct.

• Team D: And so they wanna do basically the bare minimum that's gonna look good and then effectively improve the [issue]. So we were wondering … if we just build between every other rafter, how would that affect the [issue]?

• Professor: Yeah. It's just a reduction of [this parameter]. Let me just reiterate what I said before because I noticed that when you had the source, you had the source in one direction, right? Okay. So if you put the source that goes in all directions, then you will see that, in reality, the walls are the ones that are causing all the trouble for you … So if you're asking me, without measurements, based on my measurements, all the years I have done, the walls are the most trouble makers … [Your solution] helps and contributes, but it may be overpowered by the walls before it [makes a difference].

Team D’s question, “How would that affect the issue?”, was open-ended and encouraged this engineering professor to think critically about the potential implications of Team D’s solution. Team D thus elicited feedback on their validation process and also identified additional considerations that their solution needed to address.

Some behaviors were primarily associated with specific teams. For example, Team C typically exhibited the develop mutual understanding with the stakeholder or domain expert behavior more than seven times per meeting, usually to help their user and project partners understand their design project so that they could provide more informed feedback (e.g., Team C, Phase 2 user, and partner meeting: “We've put together a couple of our preliminary ideas … put together the design requirements from the feedback you gave us. Based on that, we built a couple of [functional] prototypes we brought to show you today … they're the rough sketch of what we're thinking.”). Team C also rarely exhibited the behaviors elicit shallow responses or rigidly adhere to structure.

By comparison, Teams A, B, and D typically exhibited the behaviors elicit shallow responses (e.g., Team D, Phase 1 partner and expert meeting: “What is it like to go in the [space] for a [visitor]? Can you just tell us what that's like? Do they bring their own [materials]?”) three to six times per meeting and rigidly adhere to structure (e.g., Team A, Phase 3 partner meeting: “Okay. So the next item is options for [decoration] and I think we went through that already. Do you have any general feedback on our design, or anything general, just …”) one to two times per meeting. Teams A and B also typically exhibited the lead the stakeholder or domain expert to conclusion behavior (e.g., Team B, Phase 2 user meeting: “So, in regards to carrying [the prototype], currently it weighs 10 pounds. So, is it the weight that's the biggest challenge or is it also the size or …?”) one to two times per meeting conducted during Design Phases 1 and 2.

Teams D and E typically exhibited the introduce relevant information behavior one to two times per meeting (e.g., Team D, Phase 2 expert meeting: “So we've done a lot of preliminary searching, both online and in text books and journals and things like that, to learn about the engineering behind [this issue], and we've met with a couple of experts.”); they, along with Team A, also exhibited the explore differences between perspectives behavior one to two times per meeting (e.g., Team E, Phase 2 expert meeting: “So I think [our partners], they're hoping that this device could offer and collect some data that has substantial evidence … Is there anything that we should be looking out for in terms of creating a device that [your committee] would specifically appreciate?). Team D typically exhibited the behaviors muddle information received from the stakeholder or domain expert and introduce unclear information one to two times per meeting. Team A typically exhibited the behaviors use a student-centered meeting strategy, conflate student and stakeholder or domain expert experiences, and place own perspective above others’ one to two times per meeting.

While the teams in our study exhibited diverse approaches to conducting information gathering meetings, we observed several trends across teams. For example, participants exhibited a wide range of information gathering behaviors that aligned with recommended practices, especially during Design Phases 1 and 2. Furthermore, occurrences of information gathering behaviors that were less similar to recommended practices were generally limited across teams, although it is important to note that behaviors such as damage rapport or use a student-centered meeting strategy can negatively impact stakeholder or domain expert responses in significant ways even after a single occurrence. In addition to these positive trends, we also observed two trends across team approaches that represented specific opportunities for improvement and may reflect characteristic novice approaches to conducting information gathering meetings.

First, we observed that participants seemed to prioritize domain experts as sources of information, especially compared to other types of individuals. “Prioritization” was evident in the fact that participants employed deep exploratory behaviors such as delve into stakeholder or domain expert experiences and encourage deep thinking to a greater extent in meetings with domain experts compared to meetings with other types of individuals such as project partners. “Prioritization” was also evident in the fact that teams contacted and met with additional domain experts who had no previous affiliation with the capstone course or project sponsors (but were readily accessible through the university) to inform their projects. For example, Teams C, D, and F described conducting meetings with engineering faculty who possessed no previous connections to their projects. All users, project partners, and previous team members described by participants had either direct connections to the capstone course (e.g., users provided through the capstone course) or close relationships with project sponsors. This latter observation is unsurprising given that finding additional users or stakeholders would have required a significant time investment for most teams, particularly compared to finding additional domain experts.

Compared to meetings with domain experts, participants’ meetings with other types of individuals typically contained fewer occurrences of deep exploratory information gathering behaviors. For instance, five out of six teams met with project partners to inform their projects (Team B was the exception, since their project sponsor was a domain expert). These meetings were characterized by the behaviors build rapport with the stakeholder or domain expert and use a co-creative meeting strategy. As a reminder, “co-creative” in the context of this information gathering behavior refers mainly to meeting or project goals; teams that exhibited this behavior established space during their information gathering meetings so that stakeholders or domain experts could provide feedback that influenced the meeting and/or project direction. Teams in this study did not typically engage in co-creative ideation and solution development (e.g., as described in Refs. [11,35]) with stakeholders or domain experts, and there was not an expectation for them to do so.

As demonstrated by Team E’s excerpt in Sec. 5.3, use a co-creative meeting strategy represented an effective method for teams to solicit unstructured feedback on their design approaches and/or solution concepts. However, recommended practices for information gathering (e.g., [15,20,21]) suggest that designers should also deliberately employ exploratory questioning techniques to gather deep information about stakeholder needs and requirements to inform their understanding of their design problems and the feasibility of their solutions. The relative lack of behaviors such as delve into stakeholder or domain expert experiences and encourage deep thinking in project partner meetings indicates that participants did not consistently use deep exploratory questioning techniques with project partners. As such, even though participants did solicit feedback on their design approaches and solutions, it is unclear to what extent participants fully explored this feedback and/or gathered detailed information about their partners’ personal experiences that might inform their understandings of stakeholder needs and requirements.

While it is unclear why participants in our study infrequently employed exploratory questioning techniques to investigate the perspectives of their project partners in depth, there are a few possible explanations. For instance, Häggman et al. [37] (study involving novice engineering designers) and Sugar [58] (study involving novice software designers) observed novice designers confirming design decisions during stakeholder meetings while gathering limited additional information about stakeholders’ knowledge or experiences. Participants in our study may similarly have viewed their project partner meetings mainly as opportunities to discuss design decisions, but not necessarily as opportunities to solicit deep information related to their partners’ knowledge of their design problem. Mohedas et al. [47] also showed that some novice engineering designers struggled to interpret and apply information from stakeholders, particularly in cases when they received conflicting information from different stakeholders. As such, it is possible that our participants rarely employed behaviors such as delve into stakeholder or domain expert experiences and encourage deep thinking with project partners because they may have been unsure how to leverage deep stakeholder information effectively to inform their projects.

A related explanation is that participants may have struggled to identify specific, open-ended information goals for their meetings with project partners, especially compared to their meetings with domain experts. For example, the deep exploratory information gathering behaviors exhibited by teams during domain expert meetings seemed motivated by specific known unknowns,i.e., important information that the team knew they did not possess [2], related to technical aspects of their user requirements and/or solution concepts. It is unclear whether participants identified similarly specific and open-ended information goals for their project partner meetings as well. For instance, Team A mainly used their partner meetings to confirm, but not explore, previous design decisions, as reflected in information gathering behaviors such as use a student-centered meeting strategy. By comparison, Team E solicited open-ended feedback from their partners but rarely asked prompting questions to guide this feedback in specific directions. Without specific and open-ended information goals for project partner meetings, participants may not have felt a need to employ behaviors such as delve into stakeholder or domain expert experiences and encourage deep thinking to explore their partners’ perspectives in depth.

This hypothesized difference in information goals may also partially explain why participants in our study arranged meetings with additional domain experts, i.e., individuals who might clearly contribute relevant information to inform participants’ design decisions, but not other types of individuals whose potential contributions may have been less clear. Mohedas et al. [59] observed that novice designers in a similar capstone design context resisted meeting with individuals that they perceived as lacking expertise directly related to their design projects. Our participants may similarly have resisted meeting with stakeholders beyond their project partners and assigned users because they may have felt that such meetings would not provide project-relevant information. Alternatively, it is also possible that our participants identified specific and open-ended information goals for stakeholders but were unable to meet with individuals that they felt could provide the needed information. This latter explanation aligns with data from Mohedas et al. [47] indicating that novice designers may struggle to find stakeholders who possess the exact information that they are hoping to gather and thus conduct fewer information gathering meetings with stakeholders as a result.

In addition, we observed that participants seemed to prefer early and decisive information gathering meetings. In part, this trend was reflected in the timing of participants’ information gathering meetings, since teams in our study reported conducting most (20 out of 36 total) of their meetings during their first two design phases focusing on problem definition and concept generation/selection, respectively. In particular, Teams A (three out of four meetings), B (four out of five meetings), and D (five out of eight meetings) each conducted more than 60% of their meetings during their first two design phases. This approach to conducting information gathering meetings contrasts with recommended practices, which suggest that designers should solicit stakeholder feedback consistently throughout their projects [18,19,36,37].

The content of participants’ early information gathering meetings also seemed to reflect a preference for decisive decision-making during these meetings. All teams in this study exhibited the behaviors avoid misinterpretations, flexibly & opportunistically probe responses and verify conclusions drawn from meetings in their early meetings. The avoid misinterpretations behavior occurred less frequently in later meetings, whereas the behaviors flexibly & opportunistically probe responses and verify conclusions drawn from meetings essentially did not occur in later meetings. Teams A, B, and D, who each reported conducting more than 60% of their meetings during their first two design phases, additionally exhibited the behaviors elicit shallow responses, rigidly adhere to structure, and (for Teams A and B) lead the stakeholder or domain expert to conclusion in their early meetings.

The collection of information gathering behaviors exhibited by teams in their early meetings suggests that teams used these meetings to establish problem definitions and select preferred solution concepts as quickly as possible. Behaviors such as elicit shallow responses and lead the stakeholder of domain expert to conclusion, which do not align with recommended practices, served mainly to confirm students’ prior notions rather than invite stakeholders or domain experts to provide surprising information. In addition, participants employed the behaviors avoid misinterpretations, flexibly & opportunistically probe responses, and verify conclusions drawn from meetings primarily in situations when they sought to solidify their understandings of their design project, for instance to clarify their design requirements or verify their project sponsor’s preference for a certain solution concept. While these three behaviors do align with recommended practices—designers should consistently check that their understanding of design information aligns with the perceptions of stakeholders or domain experts [6,24,42,60] and flexibly follow up on interesting responses [8,15,42]—designers should be employing these information gathering techniques throughout their design processes. For instance, in later design stages, these techniques might facilitate the collection of comprehensive feedback related to solution prototypes that may further the designer’s understanding of their design problem [19,37]. Furthermore, stakeholder perspectives could change over the course of a semester, potentially leading stakeholder needs or requirements to change as well. Since participants mainly used the behaviors avoid misinterpretations, flexibly & opportunistically probe responses, and verify conclusions drawn from meetings to solidify their understandings of their design projects, the relative lack of these behaviors in later meetings suggests that teams did not typically revisit their understandings of their design projects over time. In other words, teams may have treated the design decisions made during their first two design phases as final, which may also explain why teams conducted fewer information gathering meetings during later design phases.

There are several reasons why participants in our study may have exhibited a preference for early and decisive information gathering meetings. For example, students were required to develop a working solution prototype by the end of the semester and likely felt that they did not have time to revisit previous design decisions and/or preserve ambiguity in their design approaches. Participants may also have tailored their efforts toward the specific design phase grading criteria, which tended to emphasize engineering analyses and design validation during later design phases.

In addition to these contextual explanations, findings from previous studies also indicate that early and decisive information gathering meetings may be characteristic of novice information gathering approaches more broadly. For instance, Crismond and Adams [61], based on a synthesis of the literature, suggested that novice designers engage in limited exploration of their design problems and start developing final solutions prematurely in their design processes. Rao et al. [62], in their study of novice engineering designers’ decision-making strategies, found that their participants were most focused on exploring user needs and characteristics during early design stages and were less focused on collecting user information during later design stages. Leahy et al. [63] observed that some novice designers fixated on their own initial ideas for solution concepts during ideation and thus devoted limited time to exploring alternative options. Lastly, Lai et al. [18] found that novice designers in another curricular design context conducted most of their information gathering meetings during the early stages of their design projects. Our findings thus align with prior observations related to the timing of novice information gathering meetings. Our findings also uniquely highlight the specific information gathering behaviors that novice engineering designers may employ during their initial information gathering meetings to help them finalize design decisions early in their design processes.

While we did not evaluate the outcomes of team projects, previous studies suggest that novice designers who conduct information gathering meetings mainly during the early stages of their design projects are less likely to produce successful design outcomes compared to novice designers who conduct meetings consistently across design stages. Atman et al. [44], in a study involving three-hour design challenges, found that novice engineering designers who gathered information only at the beginning of their problem-solving processes also generated lower quality solutions. Similarly, Häggman et al. [37], in a study of curricular design projects, observed that novice engineering designers who solicited stakeholder input only at the beginning of their concept development processes developed less successful solutions compared to peers who solicited input throughout their concept development processes. Novice designers who conduct few information gathering meetings beyond their initial design activities may also consult fewer sources of information for their projects overall, and Mohedas et al. [36] in their study of novice requirements development processes identified a positive correlation between the number of information sources that novice engineering designers consulted and the stakeholder validity of their requirements. These prior findings suggest that our participants' general approach to conducting information gathering meetings, i.e., conducting many meetings during early design phases with limited follow up during later design phases, may overall have been sub-optimal for design success, even though participants employed recommended practices for soliciting information (such as avoiding misinterpretations, flexibly & opportunistically probing responses, and verifying conclusions drawn from meetings) during their early meetings.

While the information gathering approaches of most teams in our study aligned with the two trends described above, we did note an important exception: Team C. Team C met with their users and project partners at least once during almost every design phase. These project partners had a range of backgrounds and experiences. Furthermore, Team C’s submitted meetings contained frequent instances of the behavior develop mutual understanding with the stakeholder or domain expert and few instances of information gathering behaviors that were less similar to recommended practices. Team C thus seemed to adopt an effective approach to conducting information gathering meetings: they solicited multiple perspectives consistently over time using appropriate information gathering techniques.

Notably, Team C did not possess substantially more information gathering experience than other teams in our study. The course-level support that Team C received related to conducting information gathering meetings was also very similar to the support received by Teams A and B, who were in the same capstone section. However, we did observe two key differences related to this team and their project context. First, Team C benefitted from being able to meet with their user and several different project partners simultaneously. This convenience enabled the team to solicit diverse perspectives within the context of a single meeting rather than across multiple meetings. These meetings also provided unique opportunities for Team C to resolve differences between stakeholder perspectives in real time, and this added benefit may have motivated Team C to conduct more meetings with their user and partners. In addition, as described in greater depth by Loweth et al. [45], Team C repeatedly emphasized the value of stakeholder perspectives as sources of information during their researcher interviews, especially compared to other teams. As such, the case of Team C seems to support a correlation previously theorized by Zoltowski et al. [46]: novice engineering designers who highly value stakeholder perspectives may also be more likely to solicit these perspectives consistently to inform their projects. Our findings further suggest that novice engineering designers who highly value stakeholder perspectives may also employ more effective information gathering techniques during their meetings.

One study limitation was our primary focus on meetings as a method of gathering information. There are many other ways to gather information to inform design decision-making, including observations, surveys, and academic research. Some participants may also have used email to solicit additional information from stakeholders or domain experts but did not report such email exchanges to the research team.

Teams did not submit recordings for every information gathering meeting that they conducted. Although there were no systematic omissions based on the meeting timing or type of individual involved in the meeting, more meeting recordings would help better define the behavioral trends that we observed across meetings. We also did not directly explore the reasons why our participants exhibited certain information gathering behaviors during their meetings, and future work might explore such rationales to better understand the trends observed across teams in this study.

Another limitation of our study was the relative lack of diversity across our participants, with 83% identifying as male and 71% identifying as White. A more diverse group of participants might have adopted different information gathering approaches compared to what we observed in our data. Since our study investigated the approaches of a select sample of teams, more work is also needed to determine the extent to which our findings are transferrable to other novice engineering design contexts.

Furthermore, the stakeholders and domain experts from whom teams gathered information were diverse in terms of race, gender, and, in the case of project partners, occupation. However, we did not explore this diversity in depth. As such, some of these individuals may have possessed specific knowledge or background experiences that did not factor into our analysis, but that may have influenced how teams conducted their information gathering meetings.

Lastly, we did not evaluate the outcomes of team projects. Many factors influence design successes and failures, and a holistic accounting of these factors was outside the scope of this work. As such, we were unable to verify whether teams that exhibited information gathering behaviors that aligned more closely with recommended practices also developed more successful design solutions.

Engineering design education researchers may use our findings to broaden their understandings of how design expertise related to gathering information may develop. Our comparative analysis of information gathering meetings conducted by six novice teams identified two trends, a prioritization of domain expert perspectives and a preference for early and decisive meetings, which had not been described in depth by previous studies and that may be characteristic of novice designers. Our study further presented a list of recommended practices for conducting information gathering meetings—solicit diverse perspectives, use appropriate information gathering techniques, and conduct multiple meetings over time—that may also be used to evaluate design expertise related to gathering information. Future studies can explore the extent to which more experienced designers exhibit the recommended practices that we have identified and/or consider how decisions related to these practices are interconnected.

Design practitioners can also use our list of recommended practices introduced in Sec. 2 to reflect on their own information gathering approaches. Although our work draws upon previous literature, we have not found another resource that concisely identifies soliciting diverse perspectives, using appropriate information gathering techniques, and conducting multiple meetings over time as recommended practices. Our background discussion also uniquely highlights the ways that these three recommended practices may relate to one another. Our list and discussion of recommended practices may thus assist designers in planning effective information gathering meetings that align with recommended practices and could also support designers’ reflection-in-action, i.e., reflection on design activities that occurs while designers are conducting these activities [64]. For example, our list of recommended practices may help designers identify and subsequently address gaps in ongoing information gathering activities.

Lastly, our findings suggest that novice designers would benefit from additional support related to setting explicit information goals and conducting beneficial information gathering meetings throughout their design processes. Part of this support could involve training to assist novice designers with identifying co-creative design roles for their stakeholders to facilitate information gathering; training materials could be based on case studies such as those presented by Coleman et al. [9] and Luck [10]. Design instructors might also leverage pedagogical tools such as the “Prototype for X” framework by Menold et al. [48], which has been shown to support novice designers in engaging stakeholders throughout their design projects. Furthermore, novice designers working in capstone design contexts may struggle to solicit diverse stakeholder perspectives and/or navigate conflicts between these perspectives. As suggested by the case of Team C, one potential solution may be for novice designers in capstone contexts to meet with multiple stakeholders during each information gathering meeting. Future training might thus support novice designers in applying effective methods for gathering information in focus group or participatory design workshop settings.

Our study explored novice engineering design team approaches to conducting information gathering meetings in three ways: the types of individuals from whom teams gathered information, the timing of their meetings, and the information gathering behaviors exhibited by teams during their meetings. Teams exhibited a range of behaviors that were more similar to recommended practices, especially during their early meetings. Most teams also exhibited relatively few occurrences of information gathering behaviors that were less similar to recommended practices during their meetings, although it is important to note that even scarce occurrences of these latter behaviors can significantly impact stakeholder or domain expert responses. Furthermore, our findings revealed two key trends across team approaches that represented opportunities for improvement and may be characteristic of how novice designers conduct information gathering meetings. First, teams seemed to prioritize domain expert perspectives, employing deep exploratory information gathering techniques during domain expert meetings and reaching out to additional domain experts to inform their projects. While most teams also met with project partners, their partner meetings contained few instances of deep exploratory behaviors compared to domain expert meetings. In addition, teams preferred to conduct early and decisive information gathering meetings. Teams conducted most of their meetings during the early stages of their design projects and, based on the information gathering behaviors exhibited during these meetings, mainly used early meetings to establish problem definitions and choose final solution concepts. Teams then conducted few meetings during later design phases. Both trends diverge from recommended practices for conducting information gathering meetings. However, we did observe one team, Team C, who followed recommended practices in their approach by soliciting diverse stakeholder perspectives consistently over their project and by employing appropriate information gathering techniques during their meetings. The in-depth descriptions of novice approaches presented in this paper can be used to develop tools and pedagogy that support novice designers in conducting effective information gathering meetings. Design practitioners may also use our list of recommended meeting practices to reflect on their approaches and to gather more comprehensive information about stakeholder needs and requirements.

The research team would like to express their gratitude to Jiangqiong Liu for her work verifying and revising the transcripts of information gathering meetings and researcher interviews.

• This material is based upon work supported by the National Science Foundation under Grant No. 1611687.

There are no conflicts of interest.

The authors attest that all data for this study are included in the paper. Data provided by a third party are listed in Acknowledgment.