MindSET podcast debuts with a dip into AI and ML
MindSET, a Leidos podcast featuring pioneers in science, engineering, and technology, premiered today with a conversation around artificial intelligence and machine learning. Ron Keesing, Leidos Vice President and Director of AI and ML, shared his experience and insights and helped separate hype vs. reality.
Keesing has a background in neuroscience and mathematics and started working on autonomous spacecraft at NASA in the early '90s. Today, he oversees the AI/ML Accelerator within the Leidos Innovations Center (LInC) and is responsible for the development and delivery of AI/ML solutions across the company. Here are some of the topics Keesing covered during his appearance:
- How and why to have realistic expectations about AI/ML
- Where opportunities lie for government agencies
- Exciting or interesting use cases for the technologies
- Key elements for successful implementation of AI/ML
- Solutions for defending against attacks on ML systems
- What's on the horizon for the field of AI/ML
Ron Keesing (00:00): Don't buy into the hype, right? AI Machine Learning isn't going to solve all your problems. It's not going to come in and transform every aspect of your business despite what it says on the cover of Time Magazine, but it is a very powerful tool that you need to understand how to use in order to succeed and understand how to bring into your business.
Bridget Bell (00:24): Welcome to MindSET, a Leidos podcast. I'm your host Bridget Bell.
Meghan Good (00:29): And I'm your host, Meghan Good. Join us as we talk with pioneers in science, engineering, and technology to understand their creative mindset and share their stories of innovation.
Bridget Bell (00:42): We're diving in with a topic that's getting a lot of attention right now. Artificial Intelligence and Machine Learning. We sat down with Ron Keesing to discuss how AI and ML are solving really complex problems.
Meghan Good (00:53): And he shared a lot of interesting examples of where AI/ML is being deployed today across sectors from defense and intelligence communities to civilian agencies and regulated industries like electric utilities.
Bridget Bell (01:06): We also got to speak with him on degrees of intelligence and how human reasoning and processing fits into AI. Believe it or not, AI does have limitations, so we got him talking about those.
Meghan Good (01:18): I really enjoyed that his background is in neuroscience and mathematics, which seems a perfect mix for AI/ML. But also, my background's in cybersecurity, so I was especially interested when the conversation turned to adversarial AI and what Ron predicts for the future. We hope you enjoy listening to our conversation and hearing Ron's mindset of innovation. Keep listening until the end when he offers advice on how to get away from the hype and be successful with AI.
Bridget Bell (01:44): Okay, let's get started... Welcome to MindSET. Today we are here with Ron Keesing, Vice President and Director of Artificial Intelligence and Machine Learning. Ron, thank you for being here with us.
Ron Keesing (02:00): Oh, it's a great opportunity. Thank you.
Bridget Bell (02:02): So, before we dive in, tell us a little bit about your background and how you came to this role.
Ron Keesing (02:08): Sure. So, I've actually been working in AI and Machine Learning for a very long time. I started my career back in the early '90s working on autonomous spacecraft, actually, at NASA, and it's been a lifetime fascination of mine working on both natural human and Artificial Intelligence. And it's been exciting to watch the field develop over the last two decades, and especially over the last five years, move really quickly into places where we can actually apply Machine Learning to solve real world problems.
Bridget Bell (02:37): So, what is Artificial Intelligence and Machine Learning? Let's start with the basics.
Ron Keesing (02:41): Artificial Intelligence is the development of systems that use human-like intelligence to solve problems. Now, that's kind of a loose definition. It means many things to many people, but I like to think of it as really meaning that these systems can behave in flexible ways under uncertainty and really make decisions like humans make decisions. Machine Learning is a subfield of Artificial Intelligence, where rather than telling machines what to do explicitly, we let them learn how to behave in an intelligent manner by looking at data, by learning from data. And it's actually a very exciting technique that's led to rapid advancement in the field. Most of the systems that we're familiar with today that actually have that kind of intelligent behavior, whether it's self-driving cars or Alexa, all those applications that use to an Artificial Intelligence, are all based on Machine Learning. They're based on having the machine learn to have intelligent behavior by learning from data.
Meghan Good (03:38): So, are there limits to what Artificial Intelligence and Machine Learning can do? Because we're hearing a lot of hype right now.
Ron Keesing (03:46): Sure. We are. We're sort of, in many ways, at the peak of a hype cycle around Artificial Intelligence and Machine Learning. And it's important to recognize that because we use Machine Learning, which is based on data, really, our ability to build powerful Artificial Intelligence systems is limited by problems where we have the right kind of data, and it's not just the right kind of data. That's important. The other part is a clear ability to define objectively what good behavior looks like. Because what Machine Learning does is teach a machine to minimize the difference between what it does today and what the good behavior would look like. So, for example, we've had tremendous success in fields like Go and chess where the objective quality of winning a game is really easily measured. It's actually much harder to build Machine Learning applications in problems where we can't so clearly defined a single thing to measure that means the system is performing well.
Ron Keesing (04:44): Another real big limitation is that Artificial Intelligence systems and Machine Learning systems, as we build them today, don't tend to generalize well into other domains. I mean, it's actually very interesting. We humans take knowledge we learn in one domain, and we apply it into other domains very seamlessly. But today's Artificial Intelligence systems can't do this at all. They can't draw on context or experience or data from a different domain to actually understand what's likely to happen in a new kind of problem and in a new kind of domain.
Ron Keesing (05:14): And finally, one of the other big limitations and challenges of today's systems is that because they learned from data, their performance can only be as good as the data that we give them, and it can actually learn bad things from the data as well as good. For example, one of the things that we're learning is that Machine Learning systems can learn to not only replicate, but actually amplifies certain kinds of human biases. So, if machines are actually learning to, say, make a decision about who to parole as a prisoner or who to identify as a job applicant, if the humans who create that data have some kinds of biases, cultural biases, racial biases, various kinds of prejudices, for example, the machine will not only learn to replicate those, but actually amplify them and make them worse. So, these are all kind of key limitations in the way we do AI and Machine Learning today.
Bridget Bell (06:05): It's so interesting that humans can pass their bad behaviors to AI and ML.
Ron Keesing (06:12): Yeah, it's really a profound challenge, and it's something we're working on techniques. We're funding research on how to actually build systems that can only detect when we build a system that has these biases, but automatically remove that bias from the system. But it's a challenge when you build a system like this.
Bridget Bell (06:29): So, speaking of challenges, what are the other biggest opportunities and challenges with AI/ML for government agencies today?
Ron Keesing (06:36): That's a great question, and it's one that we get from many of our customers. The first place I like to start is to think about it from the perspective of a government customer who's facing the onslaught of data that they face today. Almost all of our customers will tell us that they're getting far more data than they can possibly process or know what to do with. For example, I work with a government customer who, when we went in and help them analyze, they realized they'd only looked at less than 5% of their data ever. And faced with that kind of volume of data, they had a fundamental challenge.
Ron Keesing (07:09): Now, the beauty and the opportunity afforded by AI and Machine Learning is that you can turn that onslaught of data from a challenge into actually an opportunity because you can use all that data to actually train the machine how to improve. So, the machine can use all that data to actually learn to perform better and better if it's trained in the right ways. So, that's sort of the greatest opportunity afforded, I think, by Machine Learning is just turning the data challenge that many of our customers face on its head into an opportunity.
Ron Keesing (07:41): But I think there are a lot of other things that we have the opportunity to do using AI and Machine Learning for our customers as well. For example, there are many, many tasks that humans perform today that can be automated using techniques like robotic process automation, so we can free up human time from doing, really, not very value adding tasks like copying data from one application to another, into really valuable tasks like doing higher cognitive processing on the same information. So, a great opportunities for automation.
Ron Keesing (08:12): Now, let me tell you about a little other interesting thing you can do when you can apply automation into systems, and it comes from the program we're doing called Sea Hunter for the Navy. So, Sea Hunter is a fully autonomous vessel we've built for the Navy that's actually transited the Pacific all the way from San Diego to Hawaii completely without human intervention. One of the interesting things, it's a great, tremendous opportunity to reduce manpower on Naval vessels and operate more cheaply. But another really cool opportunity is that when you build a system like that that doesn't have human aboard, you can actually build the vessel quite differently and more efficiently than a human operated vessel would be. So, there are lots of different opportunities afforded by inserting AI and Machine Learning into real systems.
Ron Keesing (08:58): But there are challenges, too. One of the things we hear from many of our customers who've made first forays into AI and Machine Learning is that they've tried it, they've built some models, they've got some initial successes, but they're not actually generating any real business value from AI and Machine Learning. They struggle from getting past the conceptual stage, where they've shown that they have a model that can do some useful thing like predict when failures of a component are going to occur, and actually putting that into operation.
Ron Keesing (09:26): So, we, at Leidos work, with a lot of our customers on moving beyond what we call just this simple data science models into supporting a whole life cycle of AI and Machine Learning, and this is a journey for many of our customers. We have to kind of walk them through. So, it's not just about I can do a little bit of data science or build a model in a lab. It's about collecting the data, actually building something in a lab, but then moving it into production and then sustaining it through its entire life cycle. That's a challenge, I think, to any government agency that wants to deploy AI and Machine Learning, and it's typically a very large hurdle to go from what they can do in a data science environment where they just test out a concept to actually putting something out into the field, deploying it, and sustaining it. Right? That's the challenge that every government customer has, and it's something that we work with them very closely on.
Meghan Good (10:20): So, with that, Ron, you have talked about how you interact a lot with different customers, and I know within the company you interact with people who want to apply ML or AI more broadly to some of their projects. We have AI Palooza, where there's almost the showcase of those across the company. What, in your mind, is some of the most exciting uses of AI that you've heard about recently?
Ron Keesing (10:43): Well, I'll tell you one of my favorite projects that we're doing over in our civil group in the energy sector. So, this is a really interesting problem that they brought to me. I didn't really understand until they described it to me. But if you go to your local utility company and you ask them about how all of their power poles, what all the equipment is on all their power poles, they actually don't quite know the exact distribution of all of their equipment because when lineworkers go out, they move lines around and so on. So, the typical electrical utility company doesn't actually know the location of all their physical assets. So, some super smart folks in our energy practice came up with the idea of using AI and Machine Learning to actually be able to help automatically learn the whole structure of local electrical grid.
Ron Keesing (11:31): How do they do that? Well, first, we fly drones over an area and use Lidar to actually collect data on where the power poles are. When we've got that data, we then take available surface level images of power poles from things like Google street view and other sources, look on those pictures of the power poles, and actually classify all the electrical equipment on those power poles. And using these kinds of techniques with very high accuracy, we can locate where all the equipment is in a local electrical utility distribution network, and it's a great application of AI and Machine Learning to solve a really hard problem that would be incredibly human intensive to try to solve using just people driving around in trucks.
Bridget Bell (12:14): I love that example because our electric utility infrastructure is aging, and I think, like you said, utilities also don't know where they are because they're getting older and they were installed 70 years ago. And so, to be able to bring new technology to infrastructure that is decades old and find that new solution, that's a very cool example.
Ron Keesing (12:37): Yeah. And it's very current because we're looking at how the state of California right now is having to shut down huge areas of the electrical grid because they can't really monitor what's happening effectively. So, if we can use AI and Machine Learning to automate our understanding of a grid like that, we actually could potentially be able to help alleviate some of the problems that a state like California is facing.
Meghan Good (12:58): And even further, I mean, being someone who lives out kind of in the boonies, when the power goes out, we're out for a few days, right? So, if there is more knowledge that those utilities can pull from and say, "Oh, well the power is out here because of these kinds of pieces of equipment," I mean the faster time to being back up would be such a better value for me as a customer, too.
Ron Keesing (13:19): Yeah, no, it's a great application, and we're really excited about it.
Meghan Good (13:22): Really cool.
Bridget Bell (13:24): Even with these interesting applications, AI is not a magic wand, and like we talked about, there's a lot of hype around it. So, what do you see are the key elements of a successful AI system or project? What can you do to make it more successful? Move beyond the hype?
Ron Keesing (13:40): That's, again, a great question and something that we really focus on here at Leidos. Typically, when we get together with a customer and work on an engagement, the key first step is defining the right problem that you want to deploy AI and Machine Learning against. A lot of the engagements that I and my team have are actually going out and sitting down with customers and really defining the right AI Machine Learning problem, because a lot of folks will come in and tell us, "We want to use AI and Machine Learning, but we don't know what to do." Right? And the first thing is defining a good problem where you have good data available and where, and this is the most crucial part, you're going to be able to measure the value that AI and Machine Learning brings. Because a lot of folks just want to come in and try things that don't actually deliver business value. And at Leidos, we really want to focus on making sure that the AI/Machine Learning solution we bring is going to be, ultimately, delivering business value to the customer.
Ron Keesing (14:34): So, once you define that kind of right, clear objective, a business objective you want to maximize, and you figured out how you're going to deliver kind of a real return on that AI and Machine Learning investment, obviously the next step is you've got to make sure you've got a problem where data is available and where you've got the right to answer the problem. Sometimes this can be really creative. For example, figuring out ways to use maybe proxy data that's available from another source. At Leidos, we've developed a lot of solutions where we can use a combination of real and simulated data. So, there are a lot of clever ways to use data to build models, but that is really a crucial next step of the problem.
Ron Keesing (15:11): One other real challenge for everyone in this industry is making sure you've got the right talent. I think people are aware, for example, of the fact that it's pretty tough to hire great AI/Machine Learning talent, depending on who you ask. It's near impossible. On the other hand, what we find is we've got actually a lot of great Machine Learning talent within Leidos, both folks who've been doing it for a long time and really enjoy doing it at Leidos because we give them the opportunity to solve really interesting problems, but also, we've got a tremendous set of people with all the fundamental mathematical and statistical skills necessary and computer programming skills necessary. And we work on upskilling them to become expert practitioners of AI and Machine Learning because they were an applied mathematician or a statistician who's fundamental knowledge makes them eminently qualified to take up the field.
Ron Keesing (16:05): And then, we help them get up-skilled, we give them problems and projects to work on to really increase their skill level, and then we've brought them together into an AI and Machine Learning accelerator so we can kind of make them available to our customers across the company and to work on the company's most important problems using AI and Machine Learning. Because, I will say, that access to talent, to actually build the solutions, is a big bottleneck for many organizations.
Ron Keesing (16:30): And then, finally, as I talked about the, the ability to understand how to support the whole AI and Machine Learning life cycle, having kind of models for how you're going to do that, that's critical to delivering the ultimate return on investment to the customer. Because again, there's this huge hurdle between saying, "I built a model for you," and saying, "I actually put this into production, I figured out how you're going to sustain, and I figured out how you're going to update your models over their whole life cycle so that you're, ultimately, getting the real cost savings, getting the greater efficiencies, getting the higher performance that you've actually asked for in the first place."
Meghan Good (17:07): So, switching gears a little, but kind of carrying on your thought of operating and maintaining these models over time, you know that I'm a cybersecurity person, and I see AI as a new attack surface, really, within any deployed environment. What are your thoughts on protecting the models once they're out there, once they're actually manipulating data that's coming across? And actually probably the data itself, how are you thinking that we can really protect?
Ron Keesing (17:33): Yeah. It's such a critical insight that you bring there because it is absolutely another attack surface. Once you deploy AI and Machine Learning models out into the field, there's a whole new growing area of research around how you can attack these models, for different ways by attacking the data, by attacking kind of the inputs to the model and trying to get the system to do things you might want to do. We call these different kinds of attacks spoofing, for example. If you try and give the system data that might fool it, poisoning by actually giving the model during training some data that'll actually make it behave incorrectly. And now are even these interesting attacks people use to try to, if you've trained a model, to actually tried to reverse engineer the model and steal the data you use to train your model, which for example, in the world of of health, where you might use privacy protected information or intelligence where you might be using classified data to train a model, suddenly becomes this new security vulnerability.
Ron Keesing (18:33): So, we work closely with partners, and we fund our own internal research and development on ways to build certifiably robust AI and Machine Learning solutions that can detect when they're attacked and defend themselves, either by changing their behavior or by diversifying the way that they behave so that they kind of have security built in from the beginning. But it is absolutely critical, when you think about deploying into any safety critical situation, any situation where a model is going to be exposed out into the environment, that you consider the fact that people can attack these models and that there's a whole growing body of literature that makes it very easy for people to do this. They don't even have to be great scientists or researchers. There are kits you can pull right off the web that explain how to attack a Machine Learning model. And again, at Leidos, we're working on how we secure those models so that once they're deployed, they're not going to be vulnerable to those kinds of attacks.
Meghan Good (19:31): That's very cool, and I love seeing alt the traditional terms, spoofing, poisoning, from the cybersecurity field are really playing out again in this new way for a different vulnerability, too. I'm looking forward to see how that goes, this machine to machine kind of combat. The responses being taken based on the data, based on the model. And then, for all of us as almost observers of that activity happening and trying to shape it the best way that we can.
Ron Keesing (19:59): Well, I hope. I am encouraged by the fact that we're taking on, in the field of Machine Learning, we're taking on these challenges relatively early in the growth of the field. I mean, I think we all have seen around the world of, especially network security, that we all deployed out all of these models for communication and networking that really were never secured upfront. And I think in the world of AI and Machine Learning, people are talking about how can we build in the security layer early in the deployment of these solutions instead of leaving it as a bolt-on for later with all the inherent problems we've seen in the world of network protection, where we've never really achieved full protection against the range of threats.
Meghan Good (20:43): Right. And I think that's probably something it's good to consider early, and you can't be perfect. And maybe we never will be, but it's a brilliant thing to consider at this point.
Ron Keesing (20:54): Yeah.
Bridget Bell (20:55): So, AI is all about making predictions, so what are your predictions for the trends that we'll see for integration of AI and Machine Learning?
Ron Keesing (21:06): One of the trends I think we're going to see, and we already are starting to see it emerge, is a much greater focus on building systems that help humans and machines interact effectively. In the commercial world, we've already gotten used to things like talking to our Alexa and so on, but in those voice-driven systems, I think we'll see a much greater use of dialogue. Right now, current systems, you kind of have, you said, one thing to your Alexa, and it's almost like stateless, like it can't remember the last thing you said. So, I think we're seeing a much greater growth in the opportunity to have conversational technology to allow us to interact in a much more seamless and conversational way with visual interfaces. People are playing a lot with how humans can interact with AI and Machine Learning, and even, to some extent, explain how AI and Machine Learning or making decisions, or at least kind of have enough insight into how they operate, that they can interact more effectively with machines. So, I think that focus on human and machine interaction is going to be really big over the next several years.
Ron Keesing (22:07): Again, I think we're going to see a lot of focus on going, as I talked about, going that journey from, "Okay, hype, I got some AI and Machine Learning," to actually value, to how does AI and Machine Learning really deliver business value that transcends just saying, "I did it, and it's cool"? Right? To actually, what is the real business case? And then, sadly, I think it's probably not too long before we see some high profile failures associated with AI and Machine Learning. We're getting these systems out there. We've already started to see accidents from self-driving cars and other kinds of things. And I think I talked about the fact that Leidos is working on ways to protect systems against attack and so on. But frankly, there are a lot of systems being put out there that are not adequately protected, so I think it's probably just a matter of time before we see a high profile attack or we see a failure of some system that's been put into a critical situation where AI and Machine Learning is blamed for that failure. So, I think, in the short term, those are some challenges, I think, we're going to... or some trends we'll see in the field.
Ron Keesing (23:12): I think in the medium term, one of the things we're going to see kind of, and this is Meghan, towards her interest in the cyber world, is the integration of AI, Machine Learning and cyber. I mean, we already see AI and Machine Learning used to try to detect attacks or find patterns in the data. I think what we're going to see, though, is essentially AI and Machine Learning brought into the world of actually controlling our cyber defenses, because we're going to be dealing with adversaries who are using AI and Machine Learning to attack us actively, to figure out what our vulnerabilities are and constantly be attacking. And the only way we'll be able to defend ourselves is by having AI and Machine Learning actually be on the defensive side, because things will happen far too quickly for a human to control. I really think that's a big trend we'll see. And then, I also think we're going to see, in the medium term, kind of an intersection between the world of AI and Machine Learning in the world of virtual reality in some really interesting ways.
Ron Keesing (24:10): Over the longterm, it's really interesting, people talk about this problem of general AI, Artificial Intelligence that could potentially solve any kind of problem, and it's a really interesting idea. What's cool is that if you talk to researchers in AI and Machine Learning, they'll say, "Well, that's 10 years away," and when they say 10 years away what they mean is they have no idea how we're going to solve this problem." And it's still the case that if you talk to the smartest researchers in the field, they say, "It's 10 years away." Again, if you hear anybody say that, that just means we don't know yet. That's exciting, right? It means there are hard problems left to solve in this field, and it's a great field to be kind of a part of and working on because those general intelligence problems are just fascinating.
Bridget Bell (24:52): I want to go back to one of the earlier points you made, and you talked about it a couple of times, about how AI and Machine Learning is really a journey that from finding the data and implementation through actually finding the business value, so where are you seeing government agencies fall in that journey? Are there a lot that are close to finding that business value and coming through the journey, or are they all still at the beginning?
Ron Keesing (25:17): I think it really depends on the customer and how they've chosen to approach the problem. Some customers have sort of divided off AI and Machine Learning as a little problem area. They've set up a lab or set up some kind of discreet activity that's AI and Machine Learning focused. And typically what we find there is, you find some really smart people and it's really exciting and they're doing really cool work that gets published, but there's no way to flow the results of their work into real deployed solutions for their customers or for themselves. Right?
Ron Keesing (25:47): On the other hand, we have programs that are really exciting. We have one program we do for an intelligence community customer where they asked us to integrate AI and Machine Learning as part of a whole system. And in that program, we've actually deployed about 20 different Machine Learning algorithms as Machine Learning microservices that process petabytes of data in a dynamic data pipeline, and that is delivering tremendous business value that's turned kind of a data challenge that they had, from petabytes of heterogeneous data, into this powerful discovery platform where analysts can actually find any data that they're looking for using a really simple and intuitive interface all powered by AI and Machine Learning.
Ron Keesing (26:29): So, I think it really depends. We're seeing a lot of different results from our customers. Now, the good news is because we have such a broad reach across customers and markets, we as Leidos are able to bring kind of best practices from where we're seeing it being done best for government customers, as many of our government customers are ready to kind of take this journey with us, and kind of talk about best practices about what's worked and really successful deployments and about what some of the pitfalls are that kind of keep AI and Machine Learning from being deployed successfully, as well.
Meghan Good (27:00): So, you've been involved in this field, as you said, for a very long time, and I'm sure have gone through your share of the hype cycles, but also your share of what the customer challenges are and where they are. To Bridget's question, are you still finding that there are the same kind of fears or the scariness of AI, is that coming through? Or are people more in the ready to try stage, or are they past that? I would just think that as you're bringing any new technology, there's always the naysayers, right? So, where do you see where we are right now?
Ron Keesing (27:32): I think we talked to people along every stage of their feeling of threat about AI and Machine Learning. For some people, they look at it as a threat that's going to take their jobs away, right? And for others, we've got early adopters who are leaning forward and coming up with incredibly exciting and innovative ideas for how they can transform business using AI and Machine Learning. So, we work with people at every stage of that journey.
Ron Keesing (27:59): One of the things we find is that there's a model, I think, for introducing AI and Machine Learning that really works well, when you have an organization that has a lot of people across every stage. Right? And the first is, when you bring in AI and Machine Learning, for a lot of reasons, it's really good to start with a model where the AI and Machine Learning is just assisting humans do their jobs better, whether it's helping them find the data they're looking for... We don't even think about the fact that we use AI every time we use Google to go search the internet, right? That's a model where we're using the computer to help do our jobs better, right?
Ron Keesing (28:36): But if you start kind of at the simplest, where AI and Machine Learning helps me communicate more easily than having to pull down my phone and search for a song because I just told Alexa what I wanted, that's pure assistance, to a model where now maybe the Machine Learning is maybe helping me do a job differently than I could have before at a scale I couldn't before, because it augments my intelligence, right? So, I'm still really driving the system, but now I can do tasks maybe I couldn't do. Like I'm searching through hundreds of hours of video, but the machine's helping me find all the segments where interesting things happen so I can look at them and make sense of them. Right? And instead of processing eight hours of video a day, I can process hundreds of hours of the important video because that's only eight hours of interesting material for me to review.
Ron Keesing (29:23): And then, finally, into modes where systems can become truly autonomous, where we can trust that we can put a Navy vessel out into the ocean and conduct operations at sea completely unmanned, it's really a journey, and if you try to jump right to the end, then those people who are concerned and feel threatened or rightly raise concerns about the safety and security of putting these solutions out into the field immediately, you fail kind of tripping up and not succeeding. So, this multi-step process of building an AI and Machine Learning gradually, we find is really the best model that brings everyone along and actually kind of makes everyone's lives better through the application of AI and Machine Learning.
Bridget Bell (30:07): So, it's so interesting because I think, like we've talked about, some people do feel threatened by AI and ML, but as you talked through that, there's just such great opportunity for humans to be able to do their jobs better and more effectively and efficiently through these uses of technology. So, I think those examples are really interesting. Switching gears a little bit, given your broad experience, you've touched a lot of industries and areas in your life, so I'm very curious, if you were a contestant on Jeopardy, what would be that one subject that you excel in?
Ron Keesing (30:44): Well, I would have to choose either a geography or sports. I love to travel and I love to watch sports, so give me any geography question, I'll be ready.
Bridget Bell (30:55): Yeah, so just within the states, globally?
Ron Keesing (30:59): Globally. Yeah. No, I've lived overseas. Love traveling around the world whenever I can. I've been very fortunate to have the opportunity to do that. So, love to travel, love to learn, love to experience new things. Been lucky to be a lot of places, so yeah, got to go with geography.
Bridget Bell (31:14): Okay. Though, you also said sports, and I'm wondering is that like the data side of sports that we [crosstalk 00:31:20].
Ron Keesing (31:20): Well, I do love that, and I love watching a sports broadcast these days and have them pop up all kinds of data analytics. It's really cool to see the ways that the intuitions I had as a child, I used to watch a baseball game and be really frustrated that they would talk about kind of dumb statistics that made no sense in the context of the sport. So, I think there's a whole generation of geeky kids like me who grew up watching that stuff, and many of them became general managers or folks who kind of influenced the way we make decisions in sports these days. And so, it's really cool to me to watch sports and realize now they're talking about all the statistics that I was thinking about as a kid and actually kind of are natural to the field, so I enjoy that.
Meghan Good (31:59): So, I have a question for you, too. We're sitting in an office, and I wonder, since I've never worked with you in an office before, what's something surprising about how you work on a day-to-day basis that would be kind of different or something that adds some innovation to your work?
Ron Keesing (32:18): It's interesting, because I work in this field that's typically... I think people think of AI and Machine Learning being a geeky thing where people sit down in front of a computer and just code, and in fact, I spend a lot of time talking to other people. Most of the really good solutions that we come up with come from sitting down with, or actually standing up really, in front of a whiteboard with customers, with researchers. So, many people dread meetings. I actually really enjoy them because that's where most of our really great work gets done. Now, that's not the meetings where we kind of run down an agenda of boring stuff, it's one more sitting in front of a whiteboard, working hard and collaborating with other people that I find I do my absolute best work.
Bridget Bell (32:58): Really like digging in and finding the solutions to those challenges.
Ron Keesing (33:01): I love that. There's nothing I like better than being in front of a whiteboard.
Bridget Bell (33:03): That's wonderful. Well, I feel like we could spend hours talking with you on this subject. It's such an exciting topic, and your passion for it is very clear. But let's wrap up with one final question of, what advice do you have for the market?
Ron Keesing (33:18): We've talked a little bit about this already, don't buy into the hype, right? AI and Machine Learning isn't going to solve all your problems. It's not going to come in and transform every aspect of your business, despite what it says on the cover of Time Magazine. But it is a very powerful tool that you need to understand how to use in order to succeed and understand how to bring into your business. So, my advice is, choose a partner who understands how to guide you through this journey. Don't believe everything you read. Don't go to conferences and believe people who give you glossy marketing materials. Work with a partner who can show you that they've walked this journey with others and help them actually not just build something that looks cool, but actually get measurable business value from it, because that's the hardest part of succeeding in AI and Machine Learning.
Bridget Bell (34:06): Very good. Well, thank you so much for your time. It was great speaking with you today, Ron.
Ron Keesing (34:11): And great speaking to you guys. Thanks.
Bridget Bell (34:13): And thank you to our audience for listening to MindSET. If you enjoyed this episode, please share with your colleagues and visit leidos.com/mindset. And if you are specifically interested in AI/ML, please visit leidos.com/AI.