AI's Industry Impact with Jon Morra (Zefr)

What's still an open question for us is how do we talk about AI generated content that's more innocuous or more spammy and what do our customers think about that? That is something that's very very interesting. So we're working a lot on provenence detection right now and then on unsupervised clustering problems to help understand this content better. Hi, my name is Demetri Bonichi and I'm a content creator, agency owner and AI enthusiast. You're listening to the AI Agents podcast brought to you by Jot Form and featuring our very own CEO and founder Idkin Tank. This is the show where artificial intelligence meets innovation, productivity, and the tools shaping the future of work. Enjoy the show. Hello and welcome back to another episode of the AI Agents podcast. In this episode, we have a very interesting guest, the chief AI officer, John Mora, who is coming today

here from Zephr. How you doing, John? >> I'm great. Thanks so much for having me. >> Yeah, we're excited to chat. Usually, um, you know, it's interesting sometimes there's like companies that like, oh, they were an AI agent first company. There's a lot of different ways that people can get into AI as companies. And I think with with Zephr, it's going to be a pretty interesting and unique instance that we've we haven't quite had on the show before. So, you know, just to kick things off, tell us a little bit about yourself, how you got into AI um yourself and then get into the journey of um Zephyr's um forte into or not forte uh Zephr's um you know uh getting into of AI. >> Sure. So, as you said, I'm the chief AI officer here at Zephyr. I've been here about nine years. Um

and in my time here, I've led machine learning teams, data science teams, content policy teams, product teams. I've written a lot of machine learning and AI software myself. Uh so I've been through the gamut here at the company. Uh just a little bit about myself. So before this role, I was actually the director of data science at e-harmony, which back in the 2010s was a very popular internet dating brand. I think less so now. And in my time there, I was in charge of algorithmic matchmaking, dynamic pricing, fraud modeling, and churn modeling. And before that, I had my own business doing machine learning and radiation oncology. So I was very early to the game. Um and then I studied machine learning and brain MRI for uh my dissertation work. So I've been doing machine learning for a number of decades now. Um a little bit

about the company. >> So Zephr has been around since 2009. It was originally founded as Movie Clips, which was one of the biggest YouTube channels in the day. And if you ever watch >> clips of movies, then you might have watched Movie Clips. It's still a channel right now. And so the founders were these two guys, Rich and Zack. And what they realized is that if they worked with movie studios that they could license their official content and make official movie clips. Um they ended up selling that business a little bit later to Fandango and reinvested the profits into what became Zephr to do digital rights management. So the original incarnation of the company was that the movie studios would they had relationships with would come to them and say hey we love you guys but all these other people are putting out all this

unauthorized content. since you guys are experts in our content, can you go find it on YouTube and identify it and take it down? Right? So, there's a system called content ID on YouTube that they were formative in using a lot of in the early days. So, during that business, we ended up finding not only a lot of digital IP content, but we ended up making more money by not getting paid per action. So, every time we took it down, but we made more money by doing a rev share. So we'd assert ownership on behalf of the of the, you know, movie studio and then all future advertising revenue on that video would be split between Zephyr and the movie studio. And we ended up making more money on that. And then what we realized is that we were classifying a lot of content, not just

stuff with IP violation, but all kinds of content. If we could package that up and sell it to advertisers, we would make even more money because YouTube viewership was growing precipitously. And that's kind of how we got into brand advertising. In 2020, we had something kind of magical happen where Tik Tok came to us and said, "Hey, we're having a problem. A lot of big brands don't trust our content for a whole bunch of reasons. Can you come in and perform a service called verification on for brand safety? So, can you guys come in and we'll give you a feed of all the content that an advertiser is against? Use your understanding of social media data to classify that content as appropriate or not for the brand." And so we ended up winning that contract and we won. We were the first people to do

brand safety on Tik Tok. Meta came to us and asked us the same thing shortly thereafter about a year after we ended up winning that contract. So we had exclusivity for brand safety on Tik Tok and Meta. And we also did it for YouTube although we weren't the first. And so that's basically built our measurement business. So right now we have a whole business that's around understanding bad content and inappropriate content for brands and then we have another business around making sure that brands are around the right content primarily on YouTube. >> Yeah. Okay. And um I think that's that's going to be continuously um an issue for a long time and even today um it's not necessarily maybe in the same realm but you know there's there's AI content now that's coming out too that's getting better and better. I don't know if you

saw that as of this morning, maybe I'm not wrong. Uh GPT Sora 2 just got um announced the new uh did you hear about that? Yeah, >> I have heard about that. >> This is like this is like breaking news actually. So breaking the news on the interview which will come out in like a month. So just kidding. Um but no, it's uh it's crazy how that how that's happening. Yeah. So they this is a lot of uh I don't know how much of that do you necessarily uh I'm actually kind of curious what that world is like in comparison to what you're doing right cuz I don't know it's it it's weird to me that people can make all this like AI content with IP you know of >> like like you know people making like Star Wars stuff and what are your thoughts

on that whole situation and >> yeah so so obviously these generative models are getting very very good and there's not like sore too. I'm looking right now. It came out 2 hours ago. Apparently it's dropped. >> Yeah. No, it's like breaking. Yeah. It's crazy, right? Yeah. I tell people when they come on the uh the show, hey, be prepared on like stuff in your industry, but I don't expect people to know stuff two two hours ago. So, that's that's not on you, dude. If you're not chronically on LinkedIn, I don't blame you. It's fine. Yeah. >> So, so yeah. So, um but yeah. So, so definitely our customers are starting to care a lot more about the provenence of the content that's generated, whether it's generated by a person, generated by AI, updated by AI in some way. And so, we're actually doing open research

right now on how best to detect that content. And one of the we have some things that we definitely know and some things we're discovering. What we definitely know is our customers care a lot about misuse of their IP. So it's not uncommon to say stuff like big brand 123 supports XYZ political cause and then make an image that kind of indicates that and >> the brands care a lot about that. So we have a product around brand misappropriation which is a very complicated classification problem that attempts to say is there brand IP whose brand IP is it is it misused and like answer these different questions going through it. What's still an open question for us is how do we talk about AI generated content that's more innocuous or more spammy and what do our customers think about that? That is something that's very

very interesting. Um, so we're working a lot on provenence detection right now and then on unsupervised clustering problems to help understand this content better. >> Yeah. Yeah. No, that that makes a lot of sense. I hadn't considered that if you know someone were to pretend it was it was them or someone in there. Yeah. With political stuff. Yeah. And it's just wild to me because the majority of stuff I do see on these platforms are these AI generated content is is completely unique or it's just straight up having stormtroopers say weird stuff like um and you know I I can imagine that it gets it get it's getting kind of murky there. So I guess you know just to to take a step back from the AI and the big news of the day as it were um what are some of the the key

challenges that you face as a company in in this space and how do you maybe stand out from competitors who are trying to do something similar for companies? >> Yeah. Right. So basically what we do is our bread and butter is very very large scale classification. Right. So we run hundreds of millions of inferences a day and we have to do that cost effectively. We have to do that accurately for our customers. And so what our customers are demanding is they are demanding lower cost. They're demanding higher fidelity. They're demanding more classes, right? And so what we're what we're focused on is how do we deliver that at scale to our customers in a way that's repeatable and and uh defensible, right, at the end of the day. And so what we're a lot of what our cutting edge research is focused on is the

using of very very large models. So you think Gemini, GPT, Anthropic, that kind of stuff. And how do we distill down what they know to be most effective for our customers so that it meets all of their needs? >> Okay. Yeah. And you know I hundreds of that that's a lot of content first and foremost. How do you how does one do that? Not to give the secret sauce away but how does like from a high level how does how does one do that? >> Yeah. So from a very high level you start with both how much you know what your cost envelope is and how many inferences you have to run per day and then what you do is you rightsize your model in order to do that. So we have a variety of different models that we have deployed all of different parameter

counts amongst a variety of different types of GPUs. And it's basically like this giant matchmaking game of how do we most effectively deploy these GPUs so that we can use these smaller models to infer what the content's about. And then what the smaller models are able to do sometimes is they're able to promote. So if a smaller model says there's something interesting in this content, but I'm not really sure what it is, it can promote it to a higher tier where those higher tiers can be more expensive models. They can include human review and this tier promotion strategy allows us to make sure that we focus our resources and I mean that in terms of both human capital and compute capital most effectively and how many people uh once again are at Zephr >> uh right around 200. >> Okay. So like what percentage of

like human capital goes towards this type of thing at even at your own company? So we have a you know workforce that does review content right they sit all over the world right now and really the way I think about it from my standpoint is this is a giant uh incuing problem so how do we incue the work for them to do that's most impactful for the business where impactful has a lot of different meanings >> right and as we add if we were to add more people to our staff then we would just get further down that queue and that would have less impact than the top of the queue. >> Interesting. Yeah. And how over time because I'd imagine first it was it was more manual and more manual and more man or sorry it was more manual than it is now right

uh how does as a company yourself it seemed that these companies that you're helping out how have they become more and more comfortable with this idea of AI being the reviewing the the primary reviewing mechanism so to speak or like maybe the the catalyst to to have it be more effective. How how do they feel about >> So, I'm going to answer your question, but I'm going to pivot to say something else that I usually like to say in this case is we don't do AI because we want to. We do it because we have to, right? We basically our customers come to us and say we have this nuanced policy. And I'll give you an example of a policy. Um a customer says we don't want to be around crime content. And you say, "Okay, that's great. I understand that. But what about somebody

committing a crime in Grand Theft Auto audio? Is that crime content? Well, maybe. Exactly. Maybe. And some customers might have different opinions on that, right? And so like the idea is that the policy flows from what the market needs. And once we've defined the policy, then we go and say we have to implement that policy. And so there's a lot of ways you could do it. You could do it with mass human review. And in fact, we used to do a lot more human review via outsourcing, right? And this was all prel. And the reason we did it is because that's how we gather the training data necessary to train our models. But I always used to joke with my CTO, you know, years ago that it doesn't really matter how we do it as long as we can prove to ourselves and our customers

we get the right answer enough of the time. And so it just turns out that human review is not scalable. There's other much cheaper options you could do. You could do all keyword matches and just say, "Hey, here's all the keywords that matter and if it has one of these keywords, then great. And if that worked, which it doesn't, then we do that because it's cheaper." So it turns out that AI is the only solution that has both the cost and the quality that you need in order to deliver the solution. >> Yeah, you know that's a fair point. It things at some point become necessary as the market shifts and as um needs shift, right? And expectation shift. So I think AI to some extent for you guys, it just makes sense, right? Like I not not only makes sense but I don't I

feel like that's watering down the necessity. it. Well, it is necessary like you said. I'm just trying to say it in another way, but I don't need to reinvent the wheel. I would just agree it's necessary. I mean, I remember um when it came back to you're familiar with the I think we discussed this on on our first call, the pod the ad apocalypse from 2016 on YouTube. You remember that? >> 2017. Very very familiar. So, that was the catalyst for YouTube allowing thirdparty verification vendors on their platform at all. Yeah. Could you talk a little bit about why? Uh because I think I know because I was a YouTuber at the time, but explain it to the audience who maybe doesn't know. This is this is actually this is huge YouTube lore that nobody that isn't a YouTuber knows about. >> Fair enough. So

yeah, so basically YouTube would just come and say, and this is Google, and this is not only just Google, this is all platforms, right? So platforms want to be able to say, "Hey, we have three things that matter to us. We have the viewers, the people that watch the content, the creators that make the content, and the brands that pay for the content. And in a perfect world, the platform says, "We operate in a vacuum, and we figure out how to best distribute the ad impressions to the given viewers given the content in order to keep all three stool, all three legs of the stool afloat equally." Well, the brands came in and said, "Yeah, I don't know. I'm a brand that's 100 years old, 150 years old. I care a lot about my reputation and I know that if my ad is next to

unsavory content that has brand effects that won't be measured over the course of a campaign or you know a year and it could take decades for it to measure. So what happened in 2017 is the British government put out an ad for recruiting for the military. I can't remember exactly which branch. And so that ad happened to run before an ISIS recruitment video. >> Mhm. >> And this made it into not only the British tabloids but global news. And so basically the British government said, "Wait a second. Wait a second. We are not supporting we are not supporting ISIS and terrorist groups. Obviously Google, you got to fix this. We can never show up next to this." And and a lot of other brands said, "Yes, we we also cannot show up next to this type of content. Yeah. And you know what? I remember

this very plainly and this is kind of happened to a certain extent. YouTube was actually really liberal with its um monetization policies at the time in regards to how many people were able to get monetized on YouTube in comparison to probably like five, six years ago and then it actually eased up a little bit again. But you do need some a following to get monetized whereas previously pretty much anyone could just turn ads up. All right. And um to my remembrance afterwards, basically everyone was just getting like ram like uh commercials on on YouTube for a while because there was not I don't know it it was kind of it got kind of got to the point where most people weren't able to get niche product placement on on ads at the moment because they were just trying to figure it out. So I I

recall that kind of vaguely. It was a very interesting time on YouTube. But, you know, that looked to me as if that was maybe the opportunity like jumping point for you guys cuz um you know, previously like you said, they hadn't allowed third party um review platforms, right? So, not to get too deep with who you're competing against, I guess, but what um maybe what other third party platforms not are you directly facing? Not going to name any >> um but what do you do in comparison to them to to provide a better service um for you know whether what whatever it may be yet. >> So, that's a fair question. So, basically so we started we we were working with YouTube well before the ad apocalypse and at the time >> Yeah. Yeah. for years. I mean, this is all of the rights business

that we talked about was all on YouTube. We had deep relationships with YouTube for many, many, many years. And when the ad apocalypse happened, we weren't yet a verification vendor, right? Instead, we we focused on targeting. And this is, you know, many, many years ago, seven or eight years, no, more than that, years ago. And so, what we said is, hey, if you buy YouTube through us, you're just going to stay away from that content because we're not going to target that bad content that you want to be around. So we became a verification partner with YouTube a couple years later >> after we went through with Tik Tok and Meta because our customers wanted us to verify everywhere. And that kind of leads into the differentiator that we have is that we believe at least in social media that focusing on brand safety is

a key differentiator. So most of our competitors will want to do verification everywhere. So every single ad dollar you spend across every buying platform everywhere. And what we recognize is that social media is different, right? A Tik Tok video is very different than a web page. You know, a YouTube video, it's different and and a meta reels or whatever is also different. So, we built differentiated technology that was focused a lot on understanding imagery, audio, video, and text as well because that's also important in order to deliver brand suitability as a product on social media. And so, that's kind of been our differentiator against the competition. And what we found with our customers is that they tend to agree that yes, brand safety is super important in social media and they work with us on social media platforms even if they work with competitors on

other platforms. >> Yeah. >> We specifically, >> you know, it's it's hard. It's kind of hard to in the age of well, not not just AI, just in general, in the in the age of the advanced internet at this point with with security and whatnot. So like what are some of the main things? I feel like security is kind of a nefarious term or um like an amalgamist like what what does that mean practically? >> Security. >> Why did I I heard the word wrong. Sorry, not security. I'm sorry. Um just use the word safety. Yeah. What does the word safety mean? Suitability. What does safety? Yeah. Yeah. What is safe? Just Yeah. What does safety mean to them? >> So safety comes back. So, so basically there was an organization that is now defunct called the Global Alliance for Responsible Media which set out

different categorizations of ways that brands can think about content. So it had stuff like crime and adult content and profanity, terrorism, that kind of stuff. And they set out this different framework and a lot of the major brands got together and said, "Hey, this is how we want to talk about our taxonomy, >> right? And we want to use these words low, medium, and high." And then the the Garm also came out with this idea of the brand safety floor, which don't ask me why that's the most egregious because you would think it would be the least, but never mind. Um, and so when we talk about brand safety, think of it as more like no advertisers want to be around this content. So really stuff that shouldn't be on social media platforms, right? Stuff that would fail not only the monetization guidelines of social

media, but all the actual just being up, right? It shouldn't be up on on social media anywhere. um it does exist more in the open web because there's not these concrete guidelines for for the open web. But the brand suitability framework is all about what's right for a brand. This goes back to that crime example we talked about before, right? Is somebody committing a crime of Grand Theft Auto considered unsuitable for a brand? Maybe, maybe not, right? And so what we provide is this way for brands to come in and express their suitability, right? And what's happening is we have a bunch of tax we have a bunch of taxonomies that allow them to do that. Now, what we're realizing is that brands want to migrate away from taxonomies, right? We're seeing that happen. But the problem with that is that social media content is

always weirder than you think. No matter what you think it is, it's way way weirder and in different languages and different subtext and like everything you can imagine, right? Emojis and whatever. And so what we found that our value ad is being able to come to the brands with a more well- definfined recipe and say like, "Hey, when you're talking about crime or you're talking about whatever, here are the things that actually matter and they matter because we're experts in social media. We've watched the content. We we know what it is. We've done a bunch of cluster analysis on it to say this is what matters to you and still be able to take the brand's input in when they want to deviate from that in some capacity." So instead of just leaving them on their own to say describe your policy however you want,

we guide them because if they describe their policy however they want, they're not experts in social media and there will be an infinite number of corner cases. >> H yeah, that's a fair point. A lot of companies are yeah not necessarily experts in in that realm. So I think that's pretty fair. Um so you're working across a lot of these different platforms, right? You mentioned YouTube, you mentioned Meta, etc. Um, is there any specifics on any of the different platforms that you feel like is there any kind of weird idiosyncrasies between platforms on this or is it uh pretty universal or >> So there's obviously idio idiosyncrasies in the way that the platforms have content express. So for instance, Meta has like photo galleries, right? That uh Tik Tok well Tik Tok has photo galleries as well that that YouTube doesn't have for instance, right?

It's like those idiosyncrasies exist but as far as the type of content it is quite similar. So YouTube has shorts that competes with Tik Tok. Meta has reals that also competes with it. So you have your vertical formatted video, you have your longer form video, you have your photo gallery, you have your text. It is very similar and we do see a lot of crossosting. So some popular Tik Tok video will be also posted on YouTube. Yeah, I mean that that's the case, you know, with the the different mediums of content being posted on different platforms, whether it be Instagram reels and, you know, uh I guess Facebook has its own shorts kind of function now, Tik Tok has is is just that um and YouTube has its own. Actually, just a curious question about Meta itself, right? Do they even have different policies uh

in regards to in general on how it works on Instagram versus Facebook or is it pretty universal with Meta in general? >> I'm pretty sure it's universal although all the platforms do publish a uh their policies transparently. So, you know, I I don't have it handy right now, but I think it's universal. >> Okay, cool. Yeah. And what would you say, you know, with what you're doing, um, is helping combat misinformation. That's something that you even have as a landing page on your on your website. Just wanted you to kind of speak to that and why it matters, you know. I think um >> Yes. So, basically, misinformation became a topic. I mean, it's always been a topic brands care about, but before the 2024 election, it it really came into the forefront, and brands very much cared about misinformation. So we actually acquired a

a small Israeli startup in 22 called Adverify that focused a lot on this information. And the reason that we ended up acquiring them was because at the time and this is true now we kind of look at all other classifications as when you have a well-written policy and a piece of content, a person who knows the policy can apply it to the content and say yes or no matching content. Um that's not true with misinformation. So in order to get misinformation right, you can't just say is this true or false. you have to actually do work and find facts that support whether something is true or not. And so I remember talking to the CEO and he said, "Well, how would you build this?" I said, "Well, what I'd probably do is I'd go and integrate with a whole bunch of fact checkers because I

don't know the truth. The policy team doesn't know the truth of everything in the world. So we'd integrate with all these fact checkers. We'd pull down all of their facts and then we'd use them to train models to state whether something's true or not given a collection of facts. And we went and met a company out of Israel that basically did exactly that. And so it's what it's like matches made in heaven because they saw it in the same way. They said we cannot scale journalists at social media scale. We cannot have a person review every piece of content to say whether it's true or not. That just will never work. So they integrated with um the IFCN or international fact checker network and pulled down a whole bunch of facts. And so today, that's still how this process works is we have a we

have facts that are pulled in all the time and then we retrain our models given new facts regularly and we deploy those models in order to find find posts that are not true according to the consensus amongst our facteing partners. Hm. Okay. Yeah. And you don't have to, but I mean I'm I don't I'm kind of curious um you know as your companies are you know is there with this right? You know you have your own partners and stuff. So like how real time can you really make that um I guess data be in sync with with new things so to speak because you know there's so much stuff going on at once in in the zeitgeist right how do you kind of keep that like rapidly up to date >> so enter agents basically this is this is exactly where agents should enter the

conversation because this is what our customers are asking our customers are saying >> you know something just happened and not only something just happened but something just happened in my part of the world. So, you know, we do some we have some customers in Southeast Asia and unfortunately Thailand went through a war recently. Yeah. Um and >> that's big news if you live down there and less big news if you live in America. >> U and that's true anywhere. So basically what we have is is we're starting to do a lot of PC's with our customers around these agentic workflows that >> really monitor news in you know as real time as you can get and then figure out how the content that we've already discovered matches with that news what our classifications are if we need to make any changes in classifications we can

know that immediately and so I'll give you a good one that happens is unfortunately shootings happen right and it's not possible that we would know the name of a shooter or that our model would know the name of a shooter the incident it happens. That's just unfeasible, infeasible. But what we can do is know that it happened very fast and then update our decisioning to make sure that we get that kind of classification right as soon as possible. And then our customers are very interested >> in how you know people are discussing this online whether they should pause media or not. And a lot of what we're trying to do is say you don't need to pause media. you could continue to advertise through these times because Zephyr as your partner is able to find this content fast, classify, and ultimately block it. So all

of our social media partners allow us to block content that's inappropriate for advertisers. And so speed is super important there. >> Yeah. No, I can imagine that that would be that that would be the the hardest thing when you don't know about about a topic yet, like what to do in that scenario. So yeah, that that's interesting cuz you know this is kind of opening up a whole world to me mental modelwise that I had never really considered, right? Like there's um I think you mentioned and maybe I missed it. Did you mention uh X at all or no? Do you >> So no, we do not currently work with X capacity. >> Got it. Uh cuz that that would say I would say would be a platform right now where I would imagine it just be nearly impossible like to real time you know

that that platform is just like people vomiting at all times. So uh >> it is very fast and and honestly this is one of the harder problems of what we're facing with currently with agents is that it's like we can absorb all this information. What's actually tough is differentiating the signal from the noise. Like what really matters to our customers, what really matters online. And that's what's important because if we just bombard everyone with oh you know this happened and this happened and this happened we're not providing value. So this is something that we you know definitely working a lot with the development teams on with the product teams on on figuring out the right granularity here. >> Yeah. No fair. Okay. All right. So, that gives me a good idea of what your, you know, your value is that you're providing consistently and and

and I commend you for for attempting to make it make it all make sense for for our uh for people who are advertising. And it is it has got to be it has got to just be a wild landscape to deal with. So, I I couldn't even imagine getting into it. But, um you know, on the more overall practical AI side, uh I know we've we've kind of asked about you guys specifically. just kind of want to hear some of your thoughts on some more general AI based questions. um you know misinformation is a big deal obviously as you've called out and I think a good question to ask would be are you familiar with some of the recent studies or not studies but data points that have come out about how what percentage of information for like general LLM search is coming from what

websites have you heard about um any of these >> I have heard of this and I heard of a burgeoning industry like AI SEO or G geo or call it what you want where brands are trying to figure out how different models think about them relative to their competition. So, yeah, I'm familiar with that. >> Okay, cool. Yeah. What are your thoughts on how much of a percentage these general LLM search functions kind of rely on Reddit >> for their Have you seen this? >> I know that Google struck a lucrative deal for training data from Reddit. So, >> okay, interesting. Can you talk about >> This is a little while ago. Yeah, I might have missed this because I remember seeing a recent uh showcase that basically like 33%ish of like chat GBPT source data when you're using the web search function is Reddit.

>> Yes. So the the architecture we're talking about is retrieval augmented generation, right? So the LMS have a knowledge cut off date where they don't know anything about the future relative to whenever they were trained and then in order to know anything about the future they need to reach out to some third party data source and get in >> and your knowledge is only as good as what it's coming from. So yeah, if it's coming from a bunch of Reddit then that doesn't mean it's invaluable. That means that it's from Reddit and that has all the pluses and minuses of Reddit data being and the same true as of Wikipedia, right? or you could say Wikipedia is a more wellrusted source, but it's not updated as fast and this kind of thing. So, if you're trying to discover user sentiment, so you you know, you

work for a giant brand and you want to see how people think about your product, I would argue Reddit's a great place to get that data, right? But if you're looking for the truth about what happened around some event, un ideal. Yeah. Unal. Yeah. If it requires um Yeah. the large levels of journalism andor historical um accounts like I've been I read a fair amount of like heavy literature on history philosophy whatever it's so funny like do not like that kind of stuff do not trust the Google AI summary overview because it's just like okay you got to have read like hundreds of pages of of context of what people mean by words um sometimes for stuff yeah >> it's so funny because like there is all this study around the wisdom of the crowd and in fact an academic paper I used to love

to site is that if you ask a bunch of people to answer kind of an innocuous question, the wisdom of the crowd really does hold. And I think that's true for a lot of questions that people interact with LMS with. But again, when you're looking for truth, when you're looking for did something happen, it's not about the wisdom of the crowd. And having the intellectual curiosity and discipline when you're a user to know the difference is hard because it's really easy when chatbt says, "Oh, the answer to your question is this." Okay, great. Yeah. Well, it's it's fair and that's what came to mind. I haven't really asked that like hot button question so far um usually in this section, but it seems like probably were the most it would have been the most relevant for your company because >> yeah, >> them having to

deal with that in real time with news events. >> Um you know, I guess just from a job standpoint, right, where do you think the impact of what do you think the impact of AI agents are going to have on the job market kind of moving forward? you've heard some things recently about how there is a uh September surge, you know, obviously recording this on the 30th, so tomorrow would be October, but um in job growth on on LinkedIn and whatnot, and then there's people who are maybe, you know, not necessarily excited in the short-term growth prospect of jobs. What are your thoughts on how all of this will net out in the short long term with uh >> So, I think in the short term, we're still in the middle of the shakeup. I think people don't know how to use these models. They

don't know how to use them effectively. I was actually reading an interesting post by Andrew Ing recently and something I totally agree with and he says that a lot of the short-term development in AI is not going to be as much larger and larger models but it's all agentic. So it's how do we take this AI and get access to this information and these tools to perform this task and how do we do that in all the boring industries right and that's where I think a lot of opportunity is if you're an entrepreneur is bringing AI into a boring industry and automating away different parts of the jobs that don't need to be done anymore whether it's data entry or QA or whatever the customer service whatever the case may be so I think that there's a lot of short-term pain but entrepreneurial opportunity there

I think some of the long-term stuff is where it gets really interesting Right? As these models do get bigger, they do get better, they can accomplish more and they get more humanlike because right now I don't like yes they are very very very good at emulating humans but I think the next frontier for research is actually forgetfulness and lifelong learning. So these models have a knowledge cut off date. People don't have a knowledge cut off date. I'm always learning. You're always learning and we're always changing our representation of the world. We don't know how to do that yet with LOS. So once we discover that then I think it'll open a whole new can of actually replacing knowledge workers and not augmenting knowledge workers but replacing them but that still could be years off. So in the interimm what I think is the most at

risk is those activities which are repeatable which have just a set of tools that people can do and they are very much you know knowledge workers whether that's call center whether that's software development whether that's UI whatever the case may be those are the ones that are most at risk because AI agents are going to be efficient at it. Yeah, it's a fair point. I think, you know, um there's a lot of people right now that are working at jobs, especially from an associate and entry level standpoint that um I think are fortunate to be at that position now because I do think in a couple years, five probably, maybe less, I don't know. Uh there will be some issues I think for entry- level people to start finding consistent work in in certain ways. And and you know, I I don't know if you

saw as well, I don't mean to just throw daily news But 3.5 Sonnet just was released. I don't know if uh you heard yesterday it came out. >> I did see that released. Yes. >> And there was a there was a And the hard thing is with this stuff, it's always theoretical. Um the the theoretical continuous compute time multiplier was doubling every 3 to 6 months. I think something to that effect of like autonomous agent work. Yeah, I've seen it expressed as the amount of minutes an agent can work. And if like if you look at that graph over time, it's like growing, you know, somewhat exponentially. >> Yeah. And the exponential guess for the next step got obliterated yesterday. So it should have gone to 10. Um it's at it went to it was at five hours. >> Supposed to go to 10. It

went to 30. Yeah. >> Right. So, it's so interesting to me how these things come out and um you know, we're all going to have to figure out in our companies what is the way to implement these things consistently. And I guess just to ask your question, a question to your company is obviously the head of AI at your company, how are you trying to foster not only with day-to-day operations in regards to what you do as a service, but other ways, you know, like there's the little bits here and there, I'm sure. right of general work that someone does as a knowledge worker. Are you trying to impact your efficiency generally as a company with your knowledge in AI or is that not really something maybe you've focused as much on right now? >> So I actually during my IC time, my individual contributor

time, one of the things I love to do is go interview other departments and figure out what their workflows are and how I can supplement them. So I interviewed our account management recently >> and they just have a bunch they have a bunch of data that flows in from Google ads to sheets and then they make decks and everything and I was like wow I think I can really this is low hanging fruit I think I can deliver it. So the way deliver value. So the way that I deliver value is I personally will build agentic proof of concepts and deliver them to them and say hey this this works. It's not production grade. It doesn't have the security and the monitoring necessary. So like don't you know don't rely on this wholeheartedly but this is more to show what we can do with agents

and then what's effective is that those groups that get that then go back to product and say hey this is really cool and we should invest more in this and build it. Right? I like to build this flywheel. And in the full transparency, some of my experiments don't work. But that's fine. That's just my IC time and not the whole product. So, I like to deliver kind of ahead of where product is for AI agents specifically to make sure that we can derisk and make sure we're working on the right opportunities. That's how I like to work inside the company. No, I think that makes a lot of sense and yeah, that's why I was asking because it's like a lot of people don't have maybe heads of AI so to speak at um companies specifically and obviously you're you know in a unique position

and I I have figured when those type of positions will exist, right? That would be an amazing kind of opportunity is just like interviewing. And that's why I was like, "Oh, that's awesome." because I' i've been hoping that that's what people would do in that position is just going to a department be like, "All right, tell me what you're doing. Let me let me figure out how I can eic uh make this more efficient with with AI." Because I I don't expect people quite frankly to know how to be able to do automation, let alone like highlevel agentic improvement of their workflows at this point, right? I don't really think it's fair to expect that of anybody in the general workplace that doesn't tinker with this stuff like we do. So, >> you know what's so funny is that I'm hosting I have a forthcoming

panel in the next month or so where I'm hosting at a at a data conference in Southern California, Data Con LA, and the title of my p panel is is data science still a job? >> And obviously to give it away, I think I think yes, but I want to get people on there to debate. And basically the pro the the way that I think that we're going to end up is this idea that while you can anyone can call an API that calls chat GBT that's that's not hard and anyone could do that what's hard is knowing if you did a good job. So so much of the data scientist job is writing evals, making sure those eval are well formulated. They, you know, working with their product manager to make sure they express the customer belief and then doing something. We're doing something

could be prompting, it could be fine-tuning the model, it could be changing the factorization of the rag database, it could be anything in order to making sure that you're delivering that value. And so I think the role of the data science still scientist stays the same, which is we have a probabilistic problem. Everything you do with large language models is probabilistic and we have to optimize some customerf facing metric. It's just now no longer hey I'm trading a gradient boosting machine on my local machine and deploying on CPU. It's maybe I'm calling an API. Maybe I'm calling you know Amazon Bedrock or you know Google Vertex AI. But that's fine. I'm still optimizing some customerf facing outcome. >> Yeah, that makes a lot of sense. All right. Well, I appreciate you for spending the time on uh the show with me today, John. And I

guess the last thing I would just ask is what do you want to tell the audience before we close it out? Where where do you think they should go to to check you all out? Obviously there's the website, but um I bet you need anything else. Yeah. >> Yeah. So you can hit me up on LinkedIn. Um you can find me on on Zephr's website. I And yeah, I would love to talk about all things AI, all things agents. And yeah, thanks so much for having me. >> For sure. No, it was a great conversation and I'm glad we made it through the technical difficulties and uh you know, it it all worked out. So thank you so much, John, for being on the show. Please, everyone, if you could leave us a like, comment, and subscribe on the podcast on YouTube. And also make

sure to leave us a review on Apple Podcast and Spotify. That includes you, John. I'm just kidding. Um, I'm not kidding. That'd be that that help you out to be fair. Um, and uh, yeah, please make sure to check out Zephyr, and we will see you in the next one. Bye. Bye-bye. [Music]