No. Researcher will write. Yes I will work really well just to make the life of me here just recent years or so but I ration. You know really research you know what are you sure you're welcome thanks Nancy. Welcome everybody. Thanks for coming. So this is a picture from students from my class last year so I want to emphasize I don't have all the answers this is kind of how we were collaborating last year but I'm really interested in how to bring different disciplines and different people together to communicate and share information more efficiently and effectively and make better decisions. So I found that I do know that I might take the opportunity to this kind of I'm probably one of the order or system professors around I've been at this for a little while in and out of practice a little bit. I thought I'd just take a little bit of a journey through my own experience of trying to think about and develop these kinds of tools and. As I get towards the end all talk about where I see some future work as being necessary. So yes I got a master's degree at University of Illinois Chicago and this was kind of my first introduction to design computation a list. I don't list script and Autocad I was in Chicago. So I got influenced by Mr Vander Rose international style and so this was a generally a script in. Auto CAD they could. Just automatically generate these different forms based loosely on the rules of the international style. So I left there I went to work for a few years but was really intrigued by that idea of how the computer could get involved in helping us to generate and explore designs but there was no analysis on that side. It was all about generating designs and having the designer pick one and look forward so I after a few years in practice went to MIT and started to become interested in this idea of analysis and and large part because of George Stein and some other folks who were there. I became ingrained with this idea of emergence. And what does that really mean and so just as a sort of simple example here we can look at this and we can sort of see nothing we can look at this we can see a chair. We can look at this we can see window we can look at this we can see kind of both chair and window. So this functionality emerges out of the geometric form and so I was interested in career get the computer to kind of play a part in this role of interpretation and analysis as well as just generation. So I built and built a system I was actually. On a virtual collaborative network back in the mid to late ninety's. So I was pretty early with Mr Vishal lectured research lab to to develop an online collaborative system. And it kind of looked like this essential way it was a fairly so. World of blocks but we have different. Actors in the box something that looks at the world interprets webspace spatial design something looks at the world interprets inclosure something that interprets structure something that interprets light. And all sort of mediated through this central design world where each person would create their own filter or agent and then would go and and interpret and then structure structure and change the world and interpret instructional worlds. Different humans and agents could situate themselves around the world. So these are the constructors kind of on the left hand side a space constructor might interpret the need for more space and then go ahead and add elements to try to create more space for example. So I had some pretty simple management systems which would go in and look at the world and understand the evaluation from each perspective and understand who could go next and it would help the guy to design process like this I would try to improve these different perspectives. So I left on my T. and was really interested. But this was a toy problem I felt like I was a long way from having any real impact in practice I had went out to Stanford University and started a Ph D. there and. One of the first things that Martin Fisher my advisor at the time suggested I do is go out into practice and find a real problem and I think I was really great advice and something I try to do with my Ph D. students as well as get them in grained in practice to find a real problem. So the way I did that his research was around forty modeling. So I spent a lot of time gathering models from the different designers contractors bringing them together bringing in the construction schedule and constructing these kind of animated movies that would show us how for example to build the inside of the. Concert hall or in this case the outside. And so through that process. I became very aware of data flows different people's needs for different models and if you remember my mit project was all about this kind of central model around which different folks would come in and add and remove things I started to see the world a little differently here is a little bit more of a federation of loosely coupled distributed representations each of which was necessary with then dependencies between them. So on showing here is kind of the flow for the construction schedule the flow from the geometry from the different people and what was necessary to go in and build this forty model. So my thesis built on this this this idea. I called it narratives. I looked it up in a dictionary and a narrative is an accounting of events and the relationships between them. And I felt like that sort of applied for what I was thinking about here. So I developed this narrative formalism which essentially involves representations some model. Some reasoning and some dependency between that model and other models. Kind of combines our reasoning algorithms for different analyses say our representation our AI of Caesar or bends and then structuring and sequences you these interactions in terms of process. So I return to Disney Concert Hall and I got involved in the Frank Gehry building there was wasn't a right angle to be found on the entire project unfortunately so we had lots and lots of conditions of needing to build connectors between the beams and the concrete slabs things like these called Deck angles and everyone was different and my job initially was to go through and find those conditions. Manually And I said hey there. There must be a way to formalize these relationships in a way that we could do this automatically. So I took this idea of the narrative of saying these deck attachments depend on steel and concrete. And I broke it down in terms of a series of kind of lower level dependencies where I would analyze the geometry and break them down into features extrude some geometry to look for intersection between the slabs and the beings where there were intersections I would go ahead and automatically construct the deck angle attachments. This is good software prototype a kind of drag and drop think of this is kind of a predecessor to grasshopper This was back in two thousand and one two thousand and two. And so there were the kind of deck attachments which came out of the process. And did so in terms of research validation and I claim power in generality of this method by showing that on the actual concert hall they missed a whole bunch of deck attachments that we had to then go and fix in the field and my system was able to identify them automatically and for further generality I I found I thought this was kind of compelling that many of these lower level algorithms that I used to find deck attachments. I could really use in a separate different ways to do other things like in this case I was looking for cantilever conditions in the complex ceiling panel of the concert hall and was able to take many of those same algorithms as as well as some others and stitch them together in different ways to look at those conditions. So. I finished the Ph D. then hung around as a postdoc for a year or so and was able to get a job at Stanford as an assistant professor and I put together a really multi just. Structural engineers architects mechanical engineers mathematicians computer scientists. Management Science folks I got a bunch of funding from digital laboratory Boeing laboratory shouldn't be there. They didn't fund this work but they've certainly helped me since I've gotten here and work with these people on some projects I'll show you now. So you know the first thing is really identifying what the problem wasn't trying to come up with some metrics that we could start to measure our impact on our success. So we ran some case studies observations and practices the hey what's really going on out there. What does the design and decision making process look like we found this was a case study that's the one that typically there conceptual design process is when most of the major formal decisions have been made about the building authorities and that's when it all reassure things like that was happening in about five weeks highly architecturally dominated. Very few formal explicit performance objectives were being defined very little performance analysis was being performed you know basically there's a look cool and how much do we think it might cost to build and that's what they're basing the decision on. So you know kind of jumping back into theory looking at this is an analysis. Of decision methods it's became clear that every good decision requires processing a lot of information very effectively in order to look at more alternatives from more perspectives we need to do all of this much more specifically and explicitly and that we're having trouble processing all of this information. So the following tools will start to try to address this issue. Kind of different pieces from different perspectives. Of the problem. The school is called Pip or process integration platform this is work by reads and ask you. And essentially this is an online cloud based process mapping tool that allows you to actually engage in. Executing the process as well as mapping them out. So essentially projects today we typically collaborate through hierarchy's of you know F.T.P. sites or something like that what we're doing here is just adding an extra simple piece of information which is dependency between these files and folders. So if just to kind of help you interpret this each of these nodes talks about what the node is when it was built who did it and what tool is used to construct their view it. And so this is a decision making process. You know on teams objectives preferences alternatives impacts values and then each of these will break down so often that the alternative generation process is a complex process in and of itself that ends up with some rabbit models those rather models go in to daylight analysis energy analysis exciter if we dig dig into the daylight analysis. It has its own process and ultimately all this information flows up into a decision at the end there. So we use the the map to plan out the processes execute them and then you can see these colors green and red to help students understand where they were in the process. So the pip actually with this really simple additional piece of information of dependency between information started to it help us we believe communicate share and understand our processes better just in terms of the communication the collaboration I kind of are you described it. But having everybody on the same page about the flow of information and where we stand in terms of the status. With this extra piece of dependency information we can also start to search these processes. So here's an example where I'm searching for an input of some files that have the strings arc and I have C. in them and some output for strings that have L.C.A. or life cycle assessment in them so I was able to find you know a series of different processes in the database go in and copy a particular process pasted into. Our process to help you share information. And then there's this idea of understanding that we could start to develop communities around different processes start to score them talk about how long they've been they take how many times they've been used. And then we said we were also able to construct some other analyses of the database. This is a class studio project sensually this is a different project pedestrian bridge train tunnel university lab and then each row within Here is a different student. So we're basically trying to map who put out information and who consumed information. So you can see here this is a fairly tight integration of everybody sharing information with everybody in the process and down here these projects are perhaps a little more distributed not sharing information as effectively as many times really just in this case just measuring number of information handoffs nothing about the content in this case but it was interesting to the projects that were graded in terms of their just quality at the end of the actual performance of of the design and we could see at least the visual correlation between tight integration and high scores and loose integration and low scores. We ran some students given a confined design problem in a short period of time a couple of hours and had them designed with this tool half the students got PIP with all the dependencies others are asked to design with just a folder hierarchy without understanding of what the dependencies were and we found that as we coded their discussion. Sorry about that we found more trend discussions more actual iterations they were generating and analyzing more alternatives. When working in the context of the process maps their information was more consistent and they were less statements of confusion. We also tested. Sometimes if they could actually go and copy previous projects. Into their project and design in that way and we found that there were better scores of projects and less mistakes made when they were able to do that and then finally it's understanding a project that kind of just showed you being able to look at this data and perhaps try correlation between say integration and project performance. So it was really about communicating dependencies between information and getting people to to do that integration them selves. This next project is called pyro or process integration design optimization. And this is about formalizing integrating automating those those dependencies. So this is a tool called Phoenix model center something that several of you in this room. Use And here we're wrapping Frank areas digital project parametric CAD tool energy analysis doing some preprocessing for structural analysis doing structural analysis and Arabs. G.S.A. tool and then using one of. Phoenix's own genetic algorithm optimizers through our run this process so essentially generate geometry analyze it for energy analyze it for structure look at the results and decide Should you change a structural member change a material or change your commentary. So we do this around a very simple case study at first and this is the kind of results you can get three looked at I relocate a classroom. We can spin this any way we want to we can make that window bigger we can make the room wider longer. We can change different structural members in these different green members with different structural sizes. And you get results that look like this these each line here represents a design. And so if you follow all the lowest cost designs you can see what leads to them. Longer building lengths lead to lower cost design as well. Why is that as we made the building longer we constrain the floor area. So the building became thinner. And therefore it was a shorter span so it was cheaper just cheaper to frame and build a long thin building than a square building is what that's telling us you could see other. Places like call them sections good designs groups go through all different columns sections so it really doesn't matter. What column section you choose. You'll get to a good design. This is a predator front here showing the trade off between first cost and lifecycle cost where each dot here represents a design. And so we're really looking for designs along here it just depends on your preferences between structural and. Lifecycle performance cost uncertainty is not not considered of course but where I find we are often in the design practices we're back here. We're not optimal anywhere we're sort of floating around a non-optimal space. And these tools can help us get to that printer front. So the couple students dig in deep on this some bend welly when after a thermal analysis the late and energy. Analysis and did a lot of work in terms of really working through this data flow so you can model in an parametric model in digital project. And have your files prepared and push through energy analysis and daylight analysis automatically. So you can actually optimize for the US and for us figure Phleger looked at the structural analysis and he looked at a lot of steel frames for particular stadium roofs and developed a bi level optimization method where essentially if you look at this this frame we can optimize in a number of ways we could just put different members on on these different different size members on these different members and just optimize the section size we can also optimize the shape by making the larger or our or this length larger or we can optimize the topology or we have different members. Fanning in different ways for us. Just look at the first two but found you needed different optimizers for each to do sizing you needed. It's really. Called a discrete optimizer that can look at different sections sizes discretely. With a shape optimize you want to continuous optimizer that can look through geometry continuously and both of those require different types of optimizers. So that's why I've looked at communicating design process of looking optimizing at generating lots of information we finally and we need to bring all this information together and make good decisions great if we could optimize it tends to not work out that way. There are lots of objectives which are hard to fit into an optimization scheme. Excuse me so I created a tool called resurgence. Which I see is sort of the intersection of some interesting things that are going on and in our world these days. One is social networks bringing people together and online communities around certain topics we have document management. You know Dropbox things like that places to put our information online and manage it. We have task management management actually you know this is scheduling planning keeping track of tasks and how they're being executed. And finally have decision models formal ways of bringing information together wearing it. Understanding it and making decisions with it. This is a little bit sequence of as possible. So how does research work well the first step that you can often do is you'll define your different stakeholders and you ask them to weigh their goals which goal is most important to them for example. For on this particular decision and you can see in this case what we're trying to decide is for this decision of what we should prefabricated what we should build in in place on a project for these are all the different goals that the different stakeholders are concerned with and these are the different stakeholders that we are considering. Once you go through the analysis process. You end up with view. Like this would show you overall where the value is for all of these different things that we might prefabricated So for example we could see that substations for example there's lots and lots of value here perceived for if you were to prefabricated. What's the value well we can see the brown cost savings schedule savings but then there are all these other reasons we why we might want to prefabricated to in terms of creating standardization or flexibility for the future. So this was the output of the team's analysis we can also filter this information through here I've just clicked on the designers and I've just used their call priorities to figure out which decisions would make most sense for just the designer and so I can help negotiate and come to consensus on a team so and then we'll end up with views like this which in this case is a different decision looking at what structural system to pick for that for the project and all of the reasons why they chose structural steel short span because of its speed of erection first cost impact on day lighting integration of lateral options etc. So. I argue we make the basis of a lot of our decisions right now on cost and schedule and a lot of the other things that are harder to kind of formalize and bring into the same discussion with cost and schedule gets left out of a lot of decision making processes and so we says it's just trying to formalize all that rationale and show the other reasons why a decision may or may not make sense. So I I I work with Bill McDonough a little bit early on trying to understand their flows and he created this diagram that I I quite like and I show myself every now and again just remember that this is an ongoing process. I've shown you about ten years. Work up to this point. And it's you know kind of still in the fog area but start to understand how we might use these tools to enable good form or collaborative decision making for me. So what you know what are some of the outstanding issues I'd try and transfer from Stanford here to Georgia Tech. What are some of the things that I see or are still missing and and where where do we need work. So let me just explain this kind of complex diagram for you for a moment. That the dark lines the black is that decision making process that I have for I talked about a few times already. Right. And so you go through in a Figure out of your teams are you define your goals you define your alternatives you way this information. You understand the impacts you create a value you go ahead and you build. Well we should see a particular as we start to bring all of this stuff online we should be starting to learn from from this from all of our decision processes as well as from the performance of what we're actually doing in in the world and so all of that data should start to feed back. So should we should understand you know are we actually getting the building performance that we predicted we would get. And what does that tell us about what our weight should be and what the impacts actually are as opposed to just what our tools are telling us. And if we understand this. What alternatives should we really be considering and what goal should be looking at and if we understand that who should we bring on to the design team at the beginning to help us get to this best best design. So there is kind of this. Ideally this reinforcing loop of getting better and better information by making decisions capturing that rationale as well as tracking what's going on in the real world and having that feedback in. There is there's a number of kind of other and related issues with this and I've really been working and dealing with a lot of you on all. All of these issues. So one is just the organizational aspects with the livery methods work best the different contacts to make to bring people together and actually make good decisions and there's also this soft notion of trust which quickly part is pushed I'd works on. How do you get people to really share their information in the right way through this process in order to make good decisions. There's lots of work in terms of generating alternatives. Coming up you know we sit down. We just build parametric models. Is there a method or a system system with which we should approach building these parametric models we're looking through the right spaces grammar shape grammar is and graph grammar is how do we break out of the top colleges prison I guess the parametric systems can put us in and really explore larger spaces with grammars and then rename reinvestigating so idea of expert systems and how can this help us generate better alternatives more quickly. And their issues in terms of performance in terms of model accuracy certainly Friedan Jason are working a lot on these issues to rule checking space in tax different ways to analyze our designs for things beyond just energy and structure economic analysis I know that the decision theorists like to try to take all of our objectives down to one objective which is money and help us make objectives with that So how do we take something like daylight. Or views and reduce that to a number that we can make decisions with. Up on decision making game theory bringing multiple people together how to engage them in formal decision making processes as well as the uncertainty work in these decisions and how do we how do we use that to make decisions and then finally there's this concept of integration how do we bring all this together to do this all more efficiently effectively there's arbitrament G.I.S. platforms. Lifecycle management are process based ways of connecting this all together and I'm becoming a kind of big advocate of systems engineering model based systems engineering as one discipline that's kind of step back and is trying to develop the theory and the formalisms to bring all this together into one system that we can execute. One one question as we're starting to develop more and more tools more more methods and how do we pick the right method for the right challenge. So we do have we have different challenges right we have some fairly simple challenges like what overhang should I. How big my overhangs be on the window. Some more complex decisions like what structural system for a new construction project. And then some really complicated decisions like what program massing should we have for a campus. So each of these challenges should probably require a somewhat different method. And we have different methods you know we have kind of current practice methods of what we do today. We have design thinking which tries to somewhat structure a formal design process but not terribly formal. We have more formal decisions like choosing by advantages or weight rating calculate. We have optimization norm of decision theory things like that. So the question for me is we provide some guidance or understand which method to apply in which challenge which one leads to the most effective decision making for each challenge open research question for me. So how do you answer these these questions moving forward. I've been developing a course for you Mike I'm sort of thinking of it as a curriculum. Called design space construction. And I've tapped into the vertically integrated projects. Concept of Ed quite well here and that really you know I'm a little frustrating to teach students that kind of get good. And then they go off and you start the new students and they kind of get good and they go off. So this is the idea of trying to create a multi-year project where I'm students can come in as undergraduates and hopefully continue through multiple years up through the ranks of academia and then we also want to bring in professionals into this class into this process to help us understand the challenges and give us good problems to work on. So trying to integrate computing civil engineering construction architecture other folks as well as some companies are getting involved in helping us. Do the project. So we develop some quick to help you understand how to walk through these different steps of building a design space on a simple challenge and then we bring in the designers and they apply these in practice on more complicated challenges. Not to show an example of one student project here. So this was you can many of you probably recognize the names of some of the students there. So here they're working on a Perkins and well project for a hospital facade. The challenge is essentially here's a floor plan a part of this facade to have this aerated kind of edge windows here bathroom bed and we're interested in optimizing this building form for things like comfort external views of quality functionality of space. So the first thing they do is they analyze the constraints under understand what you know what's fixed what they can change as well as what some of the drivers on the design might be so they know how what analyses to focus on. And so here we are we're just trying to understand how big this window should be Reeses how big the shadow box should be for example. They get together in teams and they develop process. Diagrams this is using system all modeling language describing a high level process which breaks down into lower level processes for how they're going to go about doing their design process and then developing what I call block diagrams or think of in this kind of schema diagram so here are looking at the hospital room which has ceilings and beds and floors in the bed has a viewing point. Etc So defining the actual data model that we're going to use in this analysis. So we've been working with grasshopper number of interesting tools emerging in this area but here is part of their model for creating more logic for creating the geometry. For laying out different analysis points for analyzing daylight. For setting materials. For analyzing view angle and cone of vision. For creating a shading system for analyzing the thermal daylight performance. One sort of thing a skill. I try to teach is you know building the right model for the right challenge. If you try to build the entire tower and do all of this analysis I just showed you you'll get nowhere but recognizing that each of these rooms are repetitive. So you can do a lot of analysis on just one room and then you hear they're just building it up into a tower so they can analyze it. Statics and so that's the overall grasshopper script. And when they run it. We're using optimizers story. So such if you remember that predator front I showed you before we're trying to block less daylight in other words get more view. We're trying to have less heat dance we kind of want to be down here. And here's our pro front showing us that at some point as we lead in more and more late more and more light. We're going to be increasing our heat gain and that's where the design team the Perkins and will design was that they brought us. And so you can see that we're starting through our optimization. To explore the space and find better performing designs. So the students then go and document. Instances of each of the design alternatives. The finding them very clearly what the parameters where the LED to them. What are the goals that we're going to measure for how we're going to measure them. What are our constraints. Gathering all that data into analysis different alternatives different goals in the performance data for each of them starting understand sensitivity like you know what are the what are the moves that are making us. Get less heat gain or more he came for example understanding the data analyzing it. And then ultimately weighing all this information making decision about what the best design is. So that's the idea of this design space construction curriculum I've really reach out to all of you to help me in that I view it as a multi. Disciplinary collaborative effort of really getting people together applying research on of these design processes and understanding what works. So I've also been working on the Condell building right across the street a labor of love and frustration. I'd say at this point but we all are familiar with the design but I just want to bring this up to highlight what I think is really important. I show you that picture at the beginning of the semester at the beginning of the lecture excuse me of all those students cobbled around a laptop. I think this is really important to develop a really flexible highly digitized space that we can quickly reconfigure and so what we're looking at is half of the first floor of the canal building will be called this flex space will have dividing partitions so we can open it up and have one big space or smaller spaces and you can see just some of the configurations that we imagine being able to. To reach. Configure and use the space in different ways to really get people connected to their information better. So I thought I just started concluded with a new chapter in my life that has been really interesting. And that is through my collaboration with Perkins Well particularly the design space construction class. And they've they separate So this decided to start looking for a director of research at Perkins and well. And so I've begun that process. And so right now if you go to Perkins and Wells website and you and you Google research or not. Google or search for research you get kind of all sorts of stuff about different research centers they've done. There is a fair amount of research going on of Perkins and well John is off. Now others are doing that as you know but the firm lacks kind of direction and clarity and focus in terms of what they're trying to do in terms of research and they see that as a real priority. I'm to try to develop that. That focus and develop better collaboration with academic institutions to further their research. So how how we kind of looking at Perkins and Will has a number of practices very diverse firm but it kind of breaks down and the way I like to think of it is these four major practices education. You know we're bring the people up workplace where we do the work health care we take care of ourselves and make us healthy and in the cities that we all involve And so we have a number of kind of sub practices K. through twelve higher education for example in each of these practices. And then we're developing channels of focus and expertise in research in a number of areas design process building technology resiliency kind of new sustainability I think health energy and water and so forth. We're looking for people internally in the firm to. Head up each of these channels and then to interact with the practice in a big way to try to create create more research and and understand research problems at the beginning and then actually implement research results at the end. So I'm I'm in the process of trying to create this organization right now. A couple things to point out here so we're looking at having what are called Channel leaders for each of these channels they all have a knowledge manager. And then we're looking at creating fellows reaching back a day to me and pull folks in to have a fellow within each of these different channels or kind of building this up slowly but that that would really help to create that collaboration that we're looking for and then this thing over here called Area research we recently created a nonprofit five a one three C. entity which allows us to collaborate on a basis in indifferent research projects to go for federal funding and and so what we see here is basically you know we could bring in money through area area with then contract out the Perkins and well as well as to other academic institutions and researchers to do the work and then we would publicize the work through area. I'm up on a process guide should've been kind of clear with all the process maps. I think got purpose one needs a lot of help in terms of structuring these processes creating our information and the formal ways that we can reuse them some trying to kind of instill this research process throughout Perkins and well which is essentially starting with the practical problem. Identifying what the existing knowledge is that we can build on. If the existing knowledge of solves a practical problem. Game over. We don't need to do any research if it doesn't. We asked for research questions. We'd. Choose research methods we design research tasks we create answers to our research questions and validate them. We claim a contribution to knowledge and relate that explicitly to the existing knowledge and then we talk about its practical impact and I see it as a very important collaboration. I mean that I think the designers in the professionals have a kind of better grip and sense of what's going on on this side. And academics you guys have a much better grip on what's going on that side. So it's about bringing these two groups together executing these processes reusing the information more effectively. So my last slide. In my role at Perkins and well I'm looking to really reach back across the freeway over here to develop a number of collaboration. With Georgia Tech and other institutions and these are just some A list of some of the projects that are you know we really started to have some conversations with folks about Perkins Well designers and Georgia Tech people working together. And I'll stop there. Forty five minutes. Thank you. Chuck. It's a good an excellent question. If you talk to Phil Harrison. He's like you know put it out there keep moving and six months it's going to be old. Anyway. So that's his perspective. If you talk to a lot. So of designers and the people doing the research. They're not so sure they see you there. It's. There's kind of two issues one is it's a competitive advantage for Perkins and well and they want to keep internal The Perkins well and sometimes they want to not even share that widely right there. It's what makes them valuable in the firm. So they are a little afraid. So that's an ongoing open question. Kind of a nice thing about area is that it's short circuits a lot of that it's a nonprofit it has to be open. It has to be public and any research that goes through there has to be published. So that's one way to kind of. And that conversation quickly and if it does go through Perkins and well then we probably have to have that conversation each time. About how the knowledge that's created is being handled. One moment if we start. Right right. Yeah right. A big issue. I guess is in developing these tools take the we sit decision to write. In there. I formalize things like that you have a team and each team consists of stakeholders design. They keepers and decision makers that level of formality allows me to create certain roles downstream that facilitate the process and make it go more quickly but I don't formalize what the specific objectives are how to measure them. And I leave that up to the team and. Still that's a fair amount of work for the team we go through and formalize all these objectives gather all their alternatives gather all the information about them weigh them make the decision and it's really hard to get them to even go through that level of additional formalization so sometimes will reuse a decision where will create a decision with a bunch of objectives and it will copy that the next decision. So those objectives are formalized if you will. So I guess my my answer is it's a bit of iterative process of working with you to find out what the right level of formalization is what you can formalize ahead of time and leverage and what you need to leave to the team. It's an ongoing struggle like we sit in isn't flying off the shelf. And so the only way I know through that is like in the design space construction curriculum try it. Iterate build models try them out part of it's just getting the design team thinking that way if they can think that way they can create the formalizations it doesn't actually take that long. But they have to be trained in doing it so you have to train the the designers and then part of it's the tool formalizing right things. One winded not very clear answer but. If you do you going in. Do you need another where is our life. Right. Sounds like a great research question and I think I you know I showed you that crazy graph of like challenges against methods and design exploration. I think there are certain decisions like you know where should we go get sushi deny that or certainly probably better handled with the loose dynamic of you know Facebook and then there are going to be other decisions which are more complicated require more structured information clearly understanding the uncertainties and the tradeoffs and all that that I think we need more from so we read well this is being filmed not that this is being filmed or it. Actually I offered the recent thing at the beginning of the process. Seminar. The couple planning a space management in the great wisdom chose not to go that way. I think we really did run into issues of different people with different goals and different priorities and you know we're six months delayed off a part of that. So I think you know it's a prototype tool it's not exactly ready for prime time. Perhaps but I think something like that could have facilitated this conversation earlier we would have had this delay. Talk. I agree and maybe it relates back to our freedom. Question What do you formalize So in in the East Asian thing for example you can certainly create a stakeholder called on our occupant. And that's kind of what we did and when we finally did use we sit in on the kid elbowing so you can include that and I completely agree. They need to be stakeholders they need to find their objectives we need to figure out how important they are versus the other objectives. For. I think so. I mean that said. So the essential thing starts with figure out. Who you care about who the team is it's a first step right. Second step is finding the objectives were the constraints and one of the goals. Third step is weighing the goals. For step is weighing the stakeholders. So at that point once you're done with those four steps we know who's involved. What they care about how much they care about it. How much we care about them. So that's those are your requirements. Sure. That's what I guess I'm showing here and I showed it here as well. Right here. SARS lot of slides right there right so here's here's the outcomes. Outcomes is the effect just the design or outcomes as they affect everybody. Missing a lot of the stuff you guys are working on in terms of uncertainty and in those issues but. Both. So we are we have. First thing is values placed on the performance metrics stakeholders how much they care about objectives. Second thing is alternatives how they perform on those objectives third thing we do what's called Weight rate and calculate when we multiply how much people care about the objective by how well an alternative performs I'm not objective and that creates a sort of proxy for value and. It's all free. Really you know. Thanks very much for coming in. Thank you very nice to see you were just looking.