[00:00:05] >> Welcome to this exciting events this afternoon and it's morning in the Bay Area so good morning for everybody joining us from the Bay Area so this is an event Sacco organized between Georgia Tech and. Berkeley and we are very happy to welcome at team from the New York Times to talk about. [00:00:28] How to use emerging technology in the service of journalism and how they've executed on the sets of New York Times and. There is a whole team but I'll a little I'll just introduce at this point. The head of the New York Times r. and d. team will introduce an interim here is the numbers. [00:00:51] And that's Mark levelly So he has a storied career in. Tech in journalism at the Boston Globe at. The Washington Post and also at n.p.r. before he joined the New York Times and he joined the New York Times about 4 years ago or even or 9 years ago sorry. [00:01:14] About 4 years ago he started this new effort at the New York Times which is this r. and d. departments and how to use technologies like photogrammetry in 3 d. reconstruction all things that we care deeply about as I could then mix. And it's great to see these tools being used in that context and we'd like to hear about. [00:01:37] How we can do better in providing these tools today and see how they're used so Mark super excited about this. And please take it away. That's thinking of Frank and everyone for having us today were super excited to show you some of things are working on so much and I'm Marco Valley and. [00:02:04] We started this Mark McKay and I started this are in the group at the Times about low over 4 years ago right before an election that we thought was going to be an eventful. And you know we go off and sort of like think about the future of journalism a bit. [00:02:21] And you know so we've grown it considerably in the intervening time it's about 2 dozen people now working in a variety of areas are going to walk you through some of these some these explorations and some of the work that's that's seen the light of day we have a we have a good roster of folks. [00:02:37] With you know representing the team today so I'll just mention or meant a lot and Mark and and that order. Ask everyone to to intrude introduce themselves Thanks Mark everyone my name is or flake sure I'm a senior engineer on the r. and d. team I work. Primarily on special journalism expirations which includes computer graphics and some computer vision. [00:03:06] And I'm really excited to be here today. Mark McKay technical economy are the team. Mix of desire and technology has been working across the difference and projects across a lot of different interest in recent tech and I'm sure you I'm on the creative director for indie I work on. [00:03:33] The envisioning and designing ways for a New York Times journalists and readers to understand and interact with the tools that we're building very happy here. And then maybe having some Internet issues do is choose time with some stuff before this so just to to mention her briefly she's a creative technologist with the team and has been specializing in not going to a lot of the 3 d. capture work that we've done particularly in the in the field with a technique called photogrammetry that we're going to show so Jim monogrammed talk about photogrammetry a little bit later on Mark has some great stuff at the intersection of computer vision and sports particularly around those estimation and or has some great stuff on the work that we're doing and 3 d. visualizations of particularly making about that work and a are as well so. [00:04:26] Before we before we get into that that the cool stuff just to do a little bit of a quick set up about how we work. And what we're doing r. and d. and even media can mean a lot of a lot of different things that there are a lot of different types of groups that news organizations set up to really sort of meet the needs of our respective organizations are really a one size fits all approach at least in this era the way that we've set this up at the at the time is to really really look at the next 2 to 3 years what do we see coming in terms of emerging areas emerging technology that we think have some tangible application in the service of journalism what's starting to feel real what are we seeing in adjacent industries and what do we think we can actually use in production in that in that timeframe and so it's very much at the Applied end of the spectrum we're really looking at ways that we can we can shape our storytelling in ways that you can at you can actually see you know I think a little bit different from our group that might be thinking you know 510 plus years out. [00:05:29] Or work we work very closely with the newsroom to make that a reality so we want to really understand what problems they're trying to solve what they what stories they want to be able to tell for our readers and. You know be able to work very closely together part of the way that we design. [00:05:45] And this is that we're not we're not really on that that treadmill of news all the time and there's a lot of storytelling innovation that happens in the newsroom proper. But you know one of the ways that we set this apart and being on the lookout a little bit further out is how we can be having this conversation the Friday before an election as opposed to being being in the weeds on an exit poll visualizations or whatever our colleagues are working on this afternoon. [00:06:13] We do a lot in that you know working closely with with other groups to identify those those problems so that so that we are going to pick. You know a space that are in the teens can can sometimes fall into of creating solutions in search of problems we want to make sure that our work is very very targeted toward what we're trying to do with the newsroom. [00:06:35] There's a lot obviously in the rapid evolution of machine learning particularly computer vision and natural language processing and that's been a big area of focus for us over the past couple of years we also look at large or larger trends like the rollout of 5 g. And you know which is though obviously a multi-year thing and try to anticipate how we can take advantage of these things as they start to become real both for reporters in the field and also readers and you know at home and on the go as well so while we're not out there installing radio equipment and things like that where we're you know we've been working for the past couple of years and understanding how we can take advantage of those capabilities as they become more widely available so there will be the work that we do is a little bit adjacent to that. [00:07:21] You know written large basically if there's a if there's a buzzword of emerging technology we've probably done some taking of the tires with it we haven't we have a pretty broad remit in that regard and so that the team that we have working on these things to has a good collection of journalists and technologists of various disciplines as well as designers and strategists to really to really run this out so we try to be pretty. [00:07:49] Functional and how we how we have to organize our work in the in the people around that work as well. One thing I wanted to touch on it just sort categorically about about how we approach these things is obviously a lot particularly in the realm of machine learning in artificial intelligence these days and terms of. [00:08:10] You know making sure that models are the. Delicate and deployed in an ethical fashion you know I think as we've worked through this over the past several years it's been you know interesting how closely these questions that are really emerging in this space align with what are pretty traditional questions about ethics and journalism as well about making sure that you understand we're sort says Our where that information is coming from but also you know protecting both the you know your sources that information and then also the readers needs and interests at the at the same time as well and so you know I think as we've worked through a lot of a lot of areas of emerging tech and some things that are somewhat controversial like facial recognition you know we've been it's been helpful to be able to do this in an in a newsroom setting for 2 reasons One is that we have you know very strong standards desk that's been around for for decades that is very accustomed to dealing with these kinds of questions things that you know we're we're trying to figure out for the 1st time whether it involves a technology or not you know that it definitely manage a lot of the exceptions in our in our news report and then too is that when we're looking for specific stories to be able to try new technologies out on we can we can kind of create a bombing box with a safe a safe space to be able to do this work where you know if we don't have to worry as much about what would be kind of be like internet scale implications of adopting a technology and we can we can make these things be really really sort of concrete and tangible So the example you're seeing on the screen here is some work that we did with the royal wedding. [00:09:53] Of the the wedding was of royals at the time I guess they're not really royals with Meghan and Harry. And working with with real time facial recognition of the attendees and being able to immediately identify these folks and you know a lot of folks are pretty famous let people know what Elton John looked like. [00:10:12] But be able to work on a feature that immediately would add lower thirds of the screen so you're able to answer the question that's formulating your head as you see George in a Clooney shop which is like why are they there. To have to be able to do that in a setting like this where we're talking about well known people and at a public event we didn't tag any of the the children it's a it's a limited set of people so we can really guard against any any dangers of false positives or things like that is an example of a way of being in the work with a technology like this where the the consequences are that any negative consequences are minimal and this helps us really be able to understand what you know how what what is happening under the hood with these things in a way without sort of. [00:11:03] Releasing a **** tool into the wild that could be used and and other nefarious ways and so. That's you know I think that's that's part of how we how we approach these things now even without even valuing the technology there you know you take an example like this we felt comfortable doing this at the time I guess 33 years ago now one of the things that we need to do as a news organization is really think through any implicit endorsement of technologies that might that might circumstance from the work that we do publicly and so you know if we were looking at you know a similar event coming up next spring let's say you know we we would definitely look at the the world as it exists and the and our understanding of the technology as it exists at that I mean we might make a different decision even if even if the technology is largely the same so long story short with that satellite view things are very much moving targets for us and being able to hook the work that we do into a newsroom I think really helps us make sure that we're doing the right things in the right ways at the right times so obviously we could we can find a hold laughter Nunan on the ethics of the law but you know I think to to be able to show some cool stuff I'm going to hand it over to to Mark to talk about some computer vision. [00:12:22] Thank you. And I think we got a lot of potential and your vision for where you've been in journalism reporting I want to walk you through one example of looking at sports and in particular. Analyzing the. Or the backplate and just to give you some context of like why the New York Times a thing that we have a long history of like. [00:12:49] Showing you how. And why they do things they do are me or how and in interactive engaging format so this what you're looking at right now is go back to the last. Is a. But a one size created from the 2019 gymnastics championship and so when you 1st photography to analyze her 14 I missed her historic triple double and begin to get frame by frame tell you like what that that that is what normally happens in a few seconds and break the ducky piece and we've been doing this for a while on the next slide you'll you'll see some words from my colleagues from 2016 where we took a smoke while actually went to a. [00:13:43] Training Center the type some like sensors to her and up our this interactive where we like walk through it so you have 3 d. was very sure. But that what else can we learn from performances without even having to like to try to extract contents of the make especially in piece in advance like what can we learn from. [00:14:08] Her actual winning performance and can be like gather more magic the minute information that we started from a new plan we thought it was at a certain way to the bank spent in your vision to grow the dimension but also around all the other parts of the technique and this and we sort of alive on the fly and. [00:14:31] I can walk you through it. Starting off with the actually capturing information we planned on approach with you the 1st of August easy to capture photos of the athletes. When they're doing their routines. So the photos are good that. You can adjust the shutter speed when we work a lot like video but often they're moving so fast that. [00:15:01] They get the exposure that the people there so we would like very accurate information from each step. Of the routine. Then once we have the photos and from multiple angles I think exacting time we acknowledge he poses. Those are visible in the frame. That information is below that's useful but it's still not enough to understand where exactly it isn't 3 d. space and because we're trying to do is alive and spare not able to calibrate assist them in fact that we're rely on our access for talkers who can be there on the day off and so the day before the week of the techniques are out of the building upon research used like standardized side and sports equipment are no objects in the in the space and then we can determine the position or memory Taishan a big camera. [00:16:14] On the day I want to have all that information be in try to use back to trying really good to be put into a 3 d. world case. And then. If there's enough we've got the thing that we work you have working for him so long and deep into 3 poses and with that are able to accurately tell you like how high that job was of the rotation and also visible I was in a frame by frame basis while the. [00:16:47] But in at that point in time let me get lucky for this is I see a pretty big challenge. That. One with still photography making sure your photos are synchronized is really tough. So we develop some hardware that basically connected the cameras together on our radio a local radio network with a synchronized time from beforehand. [00:17:20] So for the part of the earthiness when we conducted feel happier when you're a practice session because when it's not. And so that was like our 1st. Person turned out to be our last capture due to the current environs I think we were at their very last practice session. [00:17:43] Though. Than. The one we have all that media has a basically a series of 1st photos that we can synchronize the lineout. We get back to the from ish. Position certainly going to the next slide and this is what it is yeah so this is what it looks like. [00:18:10] For the final I put the quite what I would for the looping video. First part of the video is that you're scene 3 photos with the 2 depo the relay and then this is a clip. The actual 3 output. So another challenge we have. Especially within the practice the arena is that between a pro that makes a model about to picking up all these people in the background we have to like 1st tell it which codes were looking for and then we can use it to be a technique that we'd like track the distance to close the next frames that we can get that can continue and post throughout the scene. [00:18:57] So in the next 5 we are taking a look at using that. Known geometry within the scene. And basically with the balance beam we know that it's a lie and that dives to the bow in advance is it Internet internationally guided I've recorded a very basic 3 d. model of that and then. [00:19:26] When the camera set up you have to do one time calibration off where those points are in that city in a space. Of what you're seeing there and then if the camera static remains consistent throughout the entire capture of the next. We discovered a lot with those estimation that. [00:19:54] Often the poses in genetics are pretty extreme and a lot of those models are. Built on. People generally people right now walking around. Rather than. These days seems that and so we actually work with a lot of different models we actually ended up with one. Difference here from the chemical branch which actually works pretty well in most scenarios but it was very important stuff that we get accurate data so. [00:20:26] We had over approach that with our tool in our pipeline there's an opportunity for a human going and might correct the data that we get the most accurate and we can trust that. That that that this magic every given are the 3 vanilla visionary having is representative what actually happened I think is important that a lot of mission going to be retired built which is always different when when your work when you're working on the actual proving the efficiency of a model is that. [00:20:58] We want the old to be human in the loop to make sure the I've been here after it. Beginning but this was some of the information that we can get. So this is from the practice capture So basically we can take points of the post that the mission is. [00:21:21] Pulling out and then we can give you the hype about theme. And also because you rotation or. There are. Pending on the athlete we can Mike adjust the analysis. And like get it essentially bought over. We said we set this up in a way that you. Felt was pretty rapidly. [00:21:52] And right now we're exploring the cable you may be used in future coverage so we were originally targeting the Summer Olympics this year. But we're definitely looking towards what we can do. At next summer's took the Olympics but also this is one of our a larger. Research area for us so we're is we're looking at other ways that computer vision confide. [00:22:20] On video on a photo you can check on our website you could have a section we're going to be posting. Aren't you know putting on your face in this area. Next click or what you do some other working 3 d. and facial drawing or. Thanks Marc to everyone or glacier an engineer on the r. and d. team so over the last 2 years the New York Times newsroom has produced groundbreaking augmented reality experience I'm not sure if anyone had the chance to check them out but anything from recreating the Apollo 11 moon landing to using a ard to visualize the sea guideline or work in the in the spatial storytelling to me has really demonstrated journalistic potential and of space for computing. [00:23:12] And so as an r.n. the team we are further exploring this opportunity and trying to imagine what it could look like for New York Times journalism to push the boundaries of spatial storytelling so questions like How can we use data visualization to tell stories that can have happened into the what does it mean to actually use the space around you to tell stories and how might the time journalism appear as a context to layer on top of the world via phone based they are our air classes in the future. [00:23:47] And so the 1st project that I'm going to walk through. Is actually a project where the this is a project where the r. and d. team work with the graphics at the time and we help visualize data from simulation done at the Kyoto Institute of Allergy and your pet. [00:24:08] During the peak of the outbreak to get the outbreak in New York City just to the full Hicks but that during the peak of the outbreak in New York we helped visualize the cost particle simulation. And. Basically we help build. Render a pipeline that is able to visualize how far a cough could spread in a space. [00:24:34] So in order to do that I'm going to dive a little bit into the technical stuff but I feel like it's probably the right crowd. That we built a cloud pipeline that ingested the research files which were done as a Kyoto Institute of Technology in Japan and these were exploited as a files and then we had then she built a layer of infrastructure that ingested the research immunization at Pre profit preprocess it filtered it cleaned it up and packaged it into an ascii format that we can then use the 3 d. offerings softwares like my own to do the interactive article but also the newspaper version of it and this was important and crucial because we were able to optimize the delivery format of the research data into something that could be rendered in a real time engine. [00:25:29] And so this is kind of a little snippet of what the data looks like this is the data visualised in pair of you so prior to us actually processing it is a simulation sequence over time of how. Of how droplets an aerosol spread in the air you can see the heavy droplets sinking down and this was what eventually was in the story. [00:25:57] Yes So this is another example this is a. Great moment to show Jupiter notebook but this is another example of the pre-processing phase where we basically use the pre-processing face to be able to color the particle in the visualization differently by their Micra meter and so our pre-processing fades in just the data took out the diameter of each droplet and was able to put a class for each one of the droplets inside of the ascii representation such that the visual they should component could read that in visualize the particles accordingly. [00:26:40] Yet and this is sort of in the pre-processing phase because we had to work with 3 d. authoring software is that then support point lab representations we actually gift wrapped each one of the Point Cloud granulated it into a mesh such that it could be read and the story could be designed in Maya but then the point cloud would we rendered by the engine accordingly at the end so it created some really interesting shapes as you can write and this is what the visualization looks like in the story. [00:27:17] So. Just and we can. And this is what the visualization looks like in the paper. For us it was it was really important to fill the data pipeline that would be able to support the newsroom and. The interactive format and the print format. Yes So along with the story we also published in a our effect in our i o. s. app where the effect would basically visualize the c.d.c. guidelines this is very early on in the outbreak and so. [00:27:57] It actually almost felt like a real thing shooting the video for this effect felt very real it was hard to actually people to come into the frame in some scenarios and then the opposite in other scenarios so this video definitely conveys that the augmented reality layer of the story. [00:28:22] So a little bit differently the 2nd example I'm going to run through is actually an argument to the reality effect we published on Instagram so we you know we we definitely. Have an augmented reality layer and our our we all to publish augmented reality effects and Instagram and I think they cater to slightly different types of readers and so it's it's by itself is an interesting thing to talk through to this effect. [00:28:55] Visualize how air pollution levels had dropped during the peak of the coronavirus lockdown and so what we did is we built this data ingestion mechanism again that would take him 2.5 readings from Berkeley actually pre-processed and prep them for an augmented reality effect that uses a particle system to basically let you talk go between before lockdown and during the lock down to see how air pollution levels dropped in 4 different cities around the world so what you can see in this loop in video is actually visualizing New Delhi. [00:29:40] And you can kind of see that we built a way for the particle system to be able to react to the data the mass average prior to the lock down and then the mass average during the last. Yeah so we're really excited about continuing to push special journalism forward I think the things we're thinking through are how to build more data visualization pipelines how can we build simulation data ingestion pipelines how can we represent data in 3 d. And how do we push spatial journalism as a format forward. [00:30:20] And. Yes this is a sort of a part of some of the stuff that we're working on just that he's out a little bit what might be coming next we're looking into how Congress he could play a role in.