On-Device AI vs Cloud: Who Really Wins?

So given your history with um ondevice AI at Google, there's kind of a big debate right now about um massive models in the cloud versus smaller private models running locally. Um my question would then be for businesses concerned about uh data sovereignty sovereignty. Yeah, data sovereignty >> and speed, which side of history do you think kind of wins out? Will our like social AI eventually live entirely around our laptops? What what are your kind of thoughts there? >> Yeah, that's a good question. So, um and we have to separate out a few things here. So, there's um a whole field or concept around secure computation um and uh and how that could be provided in a resource effective way. Um so, think about secure enclaves uh in the cloud or trusted execution environments on device. Uh in fact I think all the major cloud providers

have something of this sort. GCP recently announced a secure computing um platform that that works with Gemini. So think about data that's encrypted not only at rest and in transit but also during computation via encrypted memory. So you can get a really really high standard of uh protection and privacy by using such products. there's probably some premium for using them. I'm not I'm not quite sure. Um so, you know, if that's the goal, um it sidesteps the ondevice versus server side question. Uh ondevice has a lot of really nice advantages. It works in disconnected mode. Now, obviously, there's only so much you can do in disconnected mode. Continua is a social product. So, if you're disconnected and can't, you know, talk to other people, then um that's probably not the most important thing. latency when you are in a sort of um cell dead zone.

It's nice to have more uh reliable latency as well when you can do computation at the edge. Um and yeah, there's a lot of information that's in a device like a phone that the more pruning of the data that you could do at the edge, the better. And so you could imagine all sorts of ways of anonymizing information at the edge. There's a whole literature around something called federated learning which is something that uh I saw up front at Google around doing decentralized training of models where you can think of it like almost like a map reduction where um in this case the map stage would be um you know the the back the gradient descent that's happening at the edges and then the reducer stage would be a sort of um aggregation of of the gradients uh in the cloud. with noise added at

the client. So think about like differential privacy. So there's, you know, some plausible deniability of any specific uh data point at the clients. Uh these are things that need to continue. There's really important reasons for um this sort of privacy work to to fully develop. Um I would say that the benefits of ondevice are just pretty huge. also hearken back to what I said earlier about the cost savings because of you're sort of like distributing the inference cost across people. Um but it's not yet at the point where it can replace um what's happening where you need a really really large model in the cloud and thankfully via technologies like secure computation um you don't have to make such a a strong privacy compromise anymore.