obsidian-vault/work/zoom-transcripts/undated O-E 2026 e5TYWRytRz2WwWSe68BSWQ/transcript.md

title, date, zoom_note_id, source

title	date	zoom_note_id	source
O-E 2026	2026-11-03	e5TYWRytRz2WwWSe68BSWQ	https://us01mynotes.zoom.us/e5TYWRytRz2WwWSe68BSWQ

O-E 2026

Zoom My Notes

Transcript

• 22:03:13 · russell holt: Having a good week? Alright, let me try to figure out where I am. Here's the agenda. Yeah, a few. Thank you for adding some items. Um… So I wanted to start by highlighting this. Quarterly survey. a quarterly browser survey if you haven't seen it. It has some interesting. results from a lot of things that we're working on. we've been working on. Versus the ratings versus other browsers. Often for Windows in particular. for ad blocking has gone way up. People seem to really like that. CPM has made some improvements there with cookie banners aren't blocked. Yeah. And too many cookies sent. So some stuff we've been working on and waiting for for a long time. Pretty exciting if you go check it out if you're interested. So, just wanted to mention that.
• 22:04:54 · FD: Yeah, and just to add, I think… What's impressive to me is that NPS has also increased during the same time.
• 22:05:00 · russell holt: Can you explain NPS, Francis? It's Net Promoter something or other.
• 22:05:02 · FD: And he had to. Yeah, net promoter score. So it's basically a a signal of loyalty. So it's basically saying how likely would you recommend this product to a friend? And… And so it's like a… It's like a substitute for retention, basically. Um, yeah, so… it's a proxy. So it's, um, since we don't actually measure retention, it's the, you know, it's like a measure that we can take. Which will proxy for retention, so… Yeah, given that… satisfaction with ad blocking and CPM and MPS all raised at the same time. I mean, we can't say for sure, but we can be fairly confident that those improvements helped improve overall satisfaction. And attention.
• 22:05:52 · russell holt: Yeah, it's gone up 9 percentage points over the past year. I think that's pretty impressive. Yeah, and ad blocking in particular has gone up more than 10. Looks like we had just over 10 points since second quarter, so… That's, um… people… 63% of people in the survey rate. ad blocking on DuckDuckGo better than other browsers they use. That's up 10 points. That's pretty… You know, presumably the work we've done has actually moved that number. Who knows for sure, but… That's what we've been working on, so…
• 22:06:44 · FD: Yeah, great stuff.
• 22:06:44 · russell holt: She feel good. She feel good about that. Um, okay, uh, Alex, why don't you… you put yourself on the agenda. Tell us what's happened… happening with the, um… with your project and help us decide what to do about it.
• 22:07:00 · Alex Martemyanov: Uh, yep, so the, uh, CPM breakage, so I've, uh, summarized… You can find the link in the description. summarize the current project states and refresh the data as of today's state. It's the first day of the last public release measurements. Um, so, um… On the… on the last week progress, uh, I found out there was, uh, some, uh… Post positive sense for error pages for Mac OS and that. lowered our metrics, uh, the 30% on failures, and it is now comparable with iOS, so it's still, uh, higher than 1%, but it's not 33%, it's 1 point. 9 of permanent breakage to our daily.
• 22:07:58 · russell holt: What, what's… What's an example of an error page here?
• 22:08:04 · Alex Martemyanov: Any any error page basically any navigation like Wi Fi is not available or. Or some other problems like domain does not exist. So on that, on that page is naturally.
• 22:08:16 · russell holt: Really? Like, the new tab page?
• 22:08:20 · Alex Martemyanov: Yep.
• 22:08:21 · russell holt: Like the new tab page?
• 22:08:23 · Alex Martemyanov: Yeah. No. Not not the new tab page. New tab page was not measured, but error pages, like, domain name is.
• 22:08:24 · russell holt: Really?
• 22:08:31 · Alex Martemyanov: Incorrect.
• 22:08:32 · russell holt: Oh, okay.
• 22:08:33 · Alex Martemyanov: So on that pages, natural ACPM won't load. And those, I used to follow the metrics. And, yeah, the the effect was around thirty thirty percent, and, that dropped, around one percent of daily users.
• 22:08:42 · russell holt: Okay.
• 22:08:52 · Alex Martemyanov: as expected. Uh, another two fixes, uh, that were a work in progress were merged, so to speak, as well, but they go, uh, to public release only. The next Monday. So, those are… kind of reproduced fixes as well. Still not the main breakage point here. And I expect some observable… expect from those, but probably it won't cross the 1% line, uh, still. And, uh, so we got this, uh, largest group, Neat and Wake, as discussed. And, uh, so basically it shows, uh, normal regular flow, and, uh, it shares around 30 to 40% of, uh, all the failures. And yeah, around 30% of permanent hands. So we need to look into the extension life cycle itself, I guess. And that is the point of today's discussion. Still there is a cleanup planned for the project because there were a lot of fixes implemented and some of them didn't work out because they were kind of fixing things synthetically reproduced but not happening in production. Uh, so, uh, I'm planning to roll… roll back the fixes that are… ineffective. And, uh, we need to clean up the data metrics, because they are… they are sending large chunks of data that we probably won't need anymore. We already collected. Everything that we needed from the current state. But I think we still are going to need some permanent metrics for the feature breakage. The one that's going to be presented in the earlier. page, so we can… Observe the spikes or other stuff. And, uh, the suggestion is to modify. the extension itself to track each lifecycle step, basically. Where do we lose those communications? Whether it's the extension not loading, or it needs not. getting to the browser or other parts of the sequence? Because there are a couple of hypotheses, although they seem not… they should not cause permanent breakage, like, for example. If the config is the… config fetch from the web extension is hanging, it will cause hanging of the whole queue, but it should be reloaded by the worker after, like, half a minute. So it's not validated, and we have no idea where exactly it breaks because other other metrics are all good. And not representative here.
• 22:11:57 · russell holt: Okay, so it sounds like…
• 22:11:57 · Alex Martemyanov: Uh, so my suggestion for…
• 22:12:00 · russell holt: Go ahead.
• 22:12:02 · Alex Martemyanov: So my suggestion for this week is to clean up the fixes that didn't work and clean up the metrics and feature flags where needed or put some feature flags to permanent state and add some extra metrics. Then we put the project on hold because we have run out of all the hypothesis and current issues we know.
• 22:12:12 · russell holt: Okay.
• 22:12:25 · Alex Martemyanov: And after we start receiving the data.
• 22:12:26 · russell holt: Well, I think you should postmortem. I think you should close it. Yeah, so this is the… um… This is a good point. Uh, to, uh…
• 22:12:35 · Alex Martemyanov: Yeah.
• 22:12:36 · russell holt: I think post-mortem it. Rather than putting it on hold, because I don't know what the trigger would be for pulling it back off hold, other than, like. you know, maybe you look at the data. So, I agree with your plan here. I think it's… let me try to make sure I understand. So, there's a bunch of cleanup to do, which seems… You know, to kind of… Things that you have tried here. are still hanging out and need to be wrapped up or deleted or whatever.
• 22:13:05 · Alex Martemyanov: Yep.
• 22:13:09 · russell holt: Okay, so there's the cleanup. Then you want to make some of the temporary metrics permanent?
• 22:13:17 · Alex Martemyanov: I guess, so make… so we got… so we would have some metric that could indicate, like, in the… some future failures, if they happen. For example, something changes in macOS.
• 22:13:19 · russell holt: Okay. Yeah, okay. Okay. And then there was something about the life cycle. I wasn't sure I totally followed that.
• 22:13:38 · Alex Martemyanov: then there is two fixes that are merged, but getting released on Monday, and, uh, we'll need an extra day, uh, for data analysis before we can post-mortem, probably next Tuesday, I guess. And, uh, there is an extent.
• 22:13:47 · russell holt: Okay. Next Tuesday, perfect.
• 22:13:52 · Alex Martemyanov: Yep. And, uh, there is, uh, this extended lifecycle metrics, so basically wipe out the metrics that we were measuring, like the surroundings, uh, the events of the browser itself, uh, but dive into the extension itself and its, uh, events that it's, uh.
• 22:13:53 · russell holt: Okay.
• 22:14:11 · Alex Martemyanov: goes through just to to get an idea where it fails, at which exactly and at which step. Because otherwise, the big the largest group is, looks normal from from the data that we have.
• 22:14:20 · russell holt: Okay. Okay, I think I follow that. Okay, that sounds good.
• 22:14:35 · Alex Martemyanov: Okay. So I will communicate this, I'll communicate it, with Max and, probably have a tech design on how to.
• 22:14:36 · russell holt: Next next Tuesday.
• 22:14:43 · Alex Martemyanov: How to do it, and uh… Yeah, we'll try to complete it this week.
• 22:14:50 · russell holt: Okay. Michal, you're still the advisor. You're on board.
• 22:14:56 · Michal Smaga: Yeah, so, yeah, we synced earlier today, and this is, like, what we discussed, so I fully approve of the plan, and I think this is, like, the most reasonable.
• 22:15:00 · russell holt: I'm good.
• 22:15:05 · Michal Smaga: Way to go to use that we can drop up everything.
• 22:15:10 · russell holt: Okay.
• 22:15:10 · Michal Smaga: Yeah, great job, Alex.
• 22:15:14 · russell holt: Awesome. Thank you, Alex.
• 22:15:14 · Alex Martemyanov: Thanks.
• 22:15:17 · russell holt: Okay. Hey, Marcus, what's happening with the top sides?
• 22:15:21 · Marcos Gurgel: Hey, yes. So let me post this link here. Uh, top sites went public to 5% starting yesterday, so… Let me share my screen. The numbers that I mean, bottom line is we need more time to make any definitive conclusions. But the numbers I want to focus for now, given that we don't even have 24 hours of credible data. are these here. Uh, so, unhandled rate between top 10K and other. So, how many top 10K were detected versus how many were unhandled? How many others were detected versus how many were unhandled? And… Right now, it differs wildly between US and EU. So, top 10K in US. for less than 24 hours gave us 16% unhandled. 40% unhandled in the EU, so I think these numbers are still gonna stabilize. I don't think we should take them as meaning anything yet. Just to give you another idea.
• 22:16:23 · russell holt: What does it mean unhandled here?
• 22:16:28 · Marcos Gurgel: Unhandled means…
• 22:16:28 · russell holt: Individual pop-ups?
• 22:16:31 · Marcos Gurgel: Yes, it means, uh… It means that… We detected a pop up in using regex, but we didn't detect it with rules or heuristics.
• 22:16:43 · russell holt: Okay.
• 22:16:45 · Marcos Gurgel: So, yeah, fell through the cracks, basically. We see them in regex, but…
• 22:16:48 · russell holt: That's not 16.
• 22:16:51 · Marcos Gurgel: Sorry, go ahead.
• 22:16:51 · russell holt: That's not 16.5% of sites, that's 16.5% of…
• 22:16:57 · Marcos Gurgel: Detected. So any page that we think was there was a pop up. For 16% of them, we didn't find a pop up effectively and we didn't remove any pop ups. And… Same for other sites. So this would be the long tail. But I'm not… Very interested in this right now. I'm more interested in the top sites. Just to give you another idea of why we shouldn't… we should ignore all the numbers that came before, uh, because everything was in internal before. These were the numbers of pop ups being detected up until yesterday. So around. 20 to 30 per day. And now we're up to, you know, 30,000. So today was only the first meaningful day of data, and the releases happened. So macOS happened in the morning in the EU. But it happened in the evening in. Uh, in the EU for iOS. So, I would… I would guess that… if we look at the US numbers for yesterday, they're going to be closer to reality, because I think a lot of people were already asleep in the EU by the time the app came out. So, what else can I… can I infer from this? And by the way, these… these panels, I'm… I'm experimenting with them, I'm trying to figure out which one gives me the more… Useful number. So far, I'm not I don't think comparing them is is. is meaningful, so comparing the rate of top 10K versus the rate of others, I think it's more important to compare them with themselves, so this panel here shows me.
• 22:18:30 · russell holt: Right.
• 22:18:39 · Marcos Gurgel: how many… it essentially shows me the same as this table. So how many detected versus how many unhandled. for each bucket. So, and this is what… This is what I think would be the ideal outcome. that would result in the least amount of work in putting CPM in maintenance, which is, we see a very low number of unhandled top 10K.
• 22:19:07 · russell holt: Yeah.
• 22:19:08 · Marcos Gurgel: And we see a fairly higher number than others. I think that that's the ideal. Outcome, but I don't know if we're good.
• 22:19:13 · russell holt: Prior than others. Other.
• 22:19:16 · Marcos Gurgel: Yeah, so the long tail would have more unhandled pop ups than the top 10K.
• 22:19:24 · russell holt: Mm-hmm. Yeah.
• 22:19:25 · Marcos Gurgel: Uh, but, I mean, regardless of how they relate, I think having this number in under 20% is… would be a good result. I don't know if, I mean… It's a dice roll at this point. Yeah.
• 22:19:37 · russell holt: Do we do we have it? Do we have the relation? Do we have it? That number? Do we understand its relationship to sites in the dataset?
• 22:19:47 · Marcos Gurgel: Uh… Let me see if I understand the question.
• 22:19:50 · russell holt: Like, are sites that are, quote, covered… That we have rules for, are they still seeing unhandled pop-ups?
• 22:19:58 · Marcos Gurgel: Right. No, we don't have that because we don't know which sites exactly people are visiting. If I understand the question correctly. So we can't correlate… Yeah. We can't correlate a domain with…
• 22:20:08 · russell holt: Oh, yeah, and…
• 22:20:12 · Marcos Gurgel: That has a rule for it. Versus one that doesn't.
• 22:20:19 · russell holt: Okay, so Conrad asks, why isn't it zero? Why shouldn't it be zero?
• 22:20:26 · Marcos Gurgel: Oh, yeah. It should. It should. It's just that I was expecting, I mean, honestly, because the total coverage is right now around 22 or 30%. I was expecting this number to be higher. So yes, it should be zero. But if it's less than the average coverage. Uh, I think we're good, because then that would mean that…
• 22:20:52 · russell holt: What percent? Do you know? Sorry. That would mean that what?
• 22:20:56 · Marcos Gurgel: Yeah, go ahead.
• 22:20:57 · russell holt: No, I want to know what it means.
• 22:20:58 · Marcos Gurgel: No, uh, because right now, uh… I think… Let me see what I think it means. It means that if this is below the general average for coverage. Then we need to focus more on the long tail. If this is hitting the average for coverage, then we need to focus more on top 10K, because.
• 22:21:17 · russell holt: Okay. But we still…
• 22:21:23 · Marcos Gurgel: Because we can get it's.
• 22:21:24 · russell holt: But we still want to get it to zero.
• 22:21:25 · Marcos Gurgel: We can get more results. Yeah.
• 22:21:27 · russell holt: Right, because every site that…
• 22:21:27 · Marcos Gurgel: Yes, absolutely.
• 22:21:30 · russell holt: You know, ranked by popularity, presumably, every site that we then be sure that we. of properly completely covering has a bigger impact on that number then. Like, what's the approach for long tail? You pick.
• 22:21:42 · Marcos Gurgel: Yeah.
• 22:21:47 · russell holt: Just, you just go down the list or you pick them at random, you pick like. You have to do a lot more work in the long tail, right, than the top sites is what I'm kind of getting at.
• 22:21:59 · Marcos Gurgel: Yeah, I agree. And to that end, even at the top sites, this question of, like, which… Which sites do we tackle first? we don't quite have the answer to that, right? But we can… we can make assumptions based on sites' popularity. But… With the current metrics, we don't have a way to see whether a specific site is causing… is skewing the numbers, right?
• 22:22:23 · russell holt: Sure, sure, but… Sure, but we have, you know, first of all, they are in the top 10K, so they're top 10K, based on our definition of that.
• 22:22:32 · Marcos Gurgel: Okay.
• 22:22:34 · russell holt: But then, you know, you can go through them in some order. Still, just doing them all is, to Conrad's point, kind of what we want to aspire to, I think.
• 22:22:39 · Marcos Gurgel: Yes. Yeah, absolutely.
• 22:22:50 · russell holt: Okay, dumb question. We send this for… Uh, do you know how many people have CPM… Disabled. And do we send the pixel for…
• 22:23:00 · Marcos Gurgel: No.
• 22:23:01 · russell holt: for when. Do we only send it when a user has the feature enabled? Sorry, that's a dumb question, but we should know that.
• 22:23:09 · Marcos Gurgel: I… I… I don't know the answer to that either, but I'm fairly certain that summary pixels only go out if the user has CPM enabled, but I'll double-check that after the meeting.
• 22:23:19 · russell holt: Okay. Okay, all right.
• 22:23:22 · Marcos Gurgel: Yeah, but I don't know the answer to that either.
• 22:23:27 · russell holt: Okay, so what's your next step here?
• 22:23:27 · Marcos Gurgel: So… So, next step on… yeah, on this data… on this data is to sit and wait, but the other side of the project, which is adding the user-centric metric, I think is gonna give us…
• 22:23:31 · russell holt: I mean, we have to wait, get more data.
• 22:23:42 · Marcos Gurgel: some more actionable numbers. Because we're gonna end up with a number of, uh… Number of pages. That a user visits per day. in the top 10K with unhandled pop-ups. So what's missing from this data right now is how often a user is seeing an unhandled pop up in the top 10K or another for that matter. So… So I'm hoping that once the rest of my project is done, we have a panel that says. Uh, number of unhandled pop-ups. per user per day. In buckets, so 0 to 5, or… 5 to 10. And then from that, we can… We can gauge… the annoyance that. these unhandled pop-ups are causing on each user, right? So if… and I think that would give us a more credible signal for attacking either top 10K or other, because if we see that top 10K is… is… Relatively low here. But it's high in number of pages, in number of unhandled per page visits. Then that contradicts what we're seeing here, right? So… Yeah, I think bottom line is, I don't know very much about this data yet. I want to sit on it, but I want to progress with the other side of the project, which is to add the per user per day metric on this.
• 22:25:29 · russell holt: Yeah, uh… But that's still, I mean. You still want to get that 6… like, if that's 16… I mean, it's that…