obsidian-vault/work/zoom-transcripts/2026-10-06 SRE Incidents 2026-10-06 21 21(GMT+6 00) SrYbgHxMSlWioRrr4PGz2Q/transcript.md

title, date, zoom_note_id, source

title	date	zoom_note_id	source
SRE Incidents 2026-10-06 21:21(GMT+6:00)	2026-10-06	SrYbgHxMSlWioRrr4PGz2Q	https://us01mynotes.zoom.us/SrYbgHxMSlWioRrr4PGz2Q

SRE Incidents 2026-10-06 21:21(GMT+6:00)

Zoom My Notes

Transcript

• 21:21:52 · JŁ: Basically, it breaks trust in our capability of fire things, even though it fires for people, it gives perception it is the same chat, while it's not. It's actually like the same prompt being sent over and over again. And so so that that that's the short context. There, there is additional issue that one of our teammates can reproduce. so when they open the chat from onboarding it. it loads for a long time. It's not something that was reported by users, but this is what they see. Like, it takes up to one minute for the chat to load, and we don't know whether it's because of his location. Like, it's Hassan, who's located in Egypt.
• 21:22:38 · JŁ: So maybe it's some kind of connectivity thing like we we don't know. But but this is what we were able to to to see why they're reproducing. But yeah, so from our findings it's not possible to to be mitigated by disabling some feature flag.
• 21:23:00 · Tomasz Patrzykat: Угу?
• 21:23:01 · JŁ: Uh, so we will need to go with a hotfix route, and with some expedite review, probably, uh… and given our experience, it it usually takes up to like half a day for the review, sometimes like even even quicker. So, as far as I'm aware.
• 21:23:14 · Tomasz Patrzykat: Угу? Он Айос. Сайт, Онли? Лайк, Ноу Одер, Платформ?
• 21:23:26 · JŁ: Yeah, yeah, yeah, definitely. Definitely, iOS. Yeah, it's a native thing.
• 21:23:28 · Tomasz Patrzykat: Дэфнетли, Окей?
• 21:23:33 · JŁ: So I…
• 21:23:33 · Tomasz Patrzykat: Он. Он, фиксинг?
• 21:23:37 · JŁ: Yeah, Alex, Alex is working on it here in the call.
• 21:23:45 · Fernando Bunn: Ангел, Кимятый Сезон?
• 21:23:45 · JŁ: Bye. Sorry? Yeah, as far as I'm aware, Alex has a fix already. We we just don't know if it if it fixes all the things like I. I know that. Bartek was was on a call before, and he managed to like reproduce it even in a in a like simpler scenario like, if if it still fixes the whole thing, then I I guess we we can.
• 21:24:15 · Tomasz Patrzykat: Угу?
• 21:24:16 · JŁ: safely say that we have a fix. So we we are just now trying to to to to check on the other thing. I I guess the the thing that Hassan found. And. Yep. Anything I missed, Alex?
• 21:24:34 · Alex Martemyanov: На стрели? Для следствие бирри бэйс?
• 21:24:55 · Tomasz Patrzykat: Фикс, райт?
• 21:24:55 · Fernando Bunn: Okay, sorry.
• 21:25:03 · Tomasz Patrzykat: Дэнни Шелфикс?
• 21:25:03 · Fernando Bunn: Go ahead, Alex.
• 21:25:04 · Alex Martemyanov: Игос?
• 21:25:06 · Fernando Bunn: Alexis, I'm just happy.
• 21:25:06 · Tomasz Patrzykat: Окей, Алекс, Окей?
• 21:25:14 · Fernando Bunn: All right, so yeah, I'm looking at the logs here. Yeah, okay. There's a step that keeps replaying, the KI answer step. Is that the thing you saw, Alex, the fix you have there?
• 21:25:21 · Tomasz Patrzykat: Дуй.
• 21:25:30 · Fernando Bunn: Apparently, it's failing to get on an exit path. Like it always keeps that state no matter what. Do you have the, um… The PR? Ready?
• 21:25:43 · Fernando Bunn: Депюар Бари?
• 21:25:45 · Fernando Bunn: Where's the fix?
• 21:25:45 · Fernando Bunn: Автофикс?
• 21:25:45 · Alex Martemyanov: Не отопляры, Все?
• 21:25:50 · Fernando Bunn: Yes, indeed.
• 21:25:50 · Fernando Bunn: Санги? Mmhm. Okay, we're gonna try it here.
• 21:26:49 · Alex Martemyanov: It was always there, but… I didn't completely get it, because one of the demos that Bartek held, he had no guided screen at all. Basically, the screen that sets the, uh, the flag. the fire onboarding was complete is is made by the. You've got Express. You've got it button pressed. And if there is no button, the flag is never set and the loop is always there.
• 21:27:23 · Alex Martemyanov: But I didn't yet understand why the screen may be missing.
• 21:27:24 · Tomasz Patrzykat: Окей?
• 21:27:30 · Alex Martemyanov: Maybe it was recent change and I'm not aware of. Yeah, and uh… On-screen path. On the before I forget, choose to restore from sync. Resume onboarding, choose Duck AI, send the prompt, and go through the fire onboarding. And after the fire… There would be no, you've got this thing, yeah, and that's why it's synchronized, exactly. Foundation. And that's why all the users start experiencing.
• 21:28:13 · JŁ: Sorry. So what was the culprit? Hello!
• 21:28:16 · Alex Martemyanov: The sync restoration path that was enabled recently, and that's why it's mentioned in the task. Uh, right. In the… in the bug task. And if you choose the sync restoration path. Uh, you… Got… now you've got this screen presented. Meaning the flag is never set. So that was the activating issue. Because the bug with the button was always there, but the users were always pressing the button. And now the button is gone, because when you…
• 21:28:45 · JŁ: Okay, and do you have a… do you do one…
• 21:28:54 · Alex Martemyanov: Choose, uh, sync, restore from sync, restore using sync. On the "Oh, before I forget" screen. Right? So probably you choose the search path and restore from using sync.
• 21:29:44 · Fernando Bunn: Yeah, I'm gonna… installing our… I'm starting our fix here. See if I can… I could do station again. Um… And also I'm spinning an agent review because I previous think that my phone is just like.
• 21:30:22 · Alex Martemyanov: Yes, uh, it seems like it. Or I forget. Mmhmm. It's not the validated… the validated path, I'm just… Copying steps found. Yeah, here in the charts. At least, as it's said by Codex.
• 21:31:50 · Tomasz Patrzykat: Эй Тэнкс?
• 21:33:05 · Fernando Bunn: What is the behavior we want here? Like, if the user did not… Tap the fire button. Should that step be… repeated, as well? Like, we never want this to happen, right? Like, it doesn't matter what the user did or did not…
• 21:33:19 · Alex Martemyanov: You… the user… The user cannot miss that step. Every other controller is locked in that state.
• 21:33:30 · Fernando Bunn: What if I… what if I kill the mid without…
• 21:33:33 · Alex Martemyanov: You would just return to the same screen.
• 21:33:36 · Fernando Bunn: Okay. All right, yeah, so I can't… I can't reproduce the issue again with Alex Pierre. Gonna try from scratch.
• 21:33:46 · Alex Martemyanov: Yeah, it's basically cleared as soon as you press the fire button.
• 21:33:50 · Fernando Bunn: Yeah, I saw the change there. But I'm trying from scratch again.
• 21:33:58 · Alex Martemyanov: Yeah, maybe the steps I just sent are not correct steps. I'm just trying to figure it out via cloud. The Sync Restore screen is called "Let's Pick Up Where You Left Off".
• 21:34:28 · Fernando Bunn: Yeah, the two things the review here is saying, but I'll let you decide if it makes sense or not. One is the one that I called before. It found the same thing, but basically… If for some reason there's another entry point. um…
• 21:34:47 · Alex Martemyanov: Yeah, it's the same thing, anyways.
• 21:34:47 · Fernando Bunn: On that screen, it's gonna miss it, yeah.
• 21:34:50 · Alex Martemyanov: Because when you get to the fire onboarding state, it sets the flag, and when you hit the fire button, it resets the flag. So that's the entry and exit.
• 21:34:50 · Fernando Bunn: And one of… And. Okay. And another one is just this saying that it's missing tests from the call points. I don't really mind this one.
• 21:34:59 · Alex Martemyanov: I would say. Yeah, it just makes the bug obvious for… it's just…
• 21:35:08 · Fernando Bunn: Okay, but…
• 21:35:12 · Alex Martemyanov: Makes the bug inevitable for all the users who use the sync feature.
• 21:35:21 · Fernando Bunn: Okay, I'm trying again. So skip. Skip. Next. Type. Bye. Maybe it's the KI.
• 21:35:35 · Alex Martemyanov: Yeah, so set to returning user. Restart, press restore my stuff. And get through the onboarding. Um, whatever. I need to…
• 21:36:03 · Fernando Bunn: Okay, seems to be… I did the steps from scratch, at least the ones that Bortek reproduced seems to be working here.
• 21:36:11 · Alex Martemyanov: Yeah, that exact scenario is reproduced, if you omit the… We've got this button, we've got the screen, but if there is no screen, yeah, it's 100% reproducible, and it's obvious, and the fix should just fix the file onboarding.
• 21:36:16 · Fernando Bunn: Mm-mm.
• 21:36:25 · Alex Martemyanov: And uh… probably it's not up to us to fix the flow itself, it's just the… the flag issue. So, should I rebase it to the current release branch? Origin, release, iOS.
• 21:36:42 · JŁ: One, all right. I was talking to Thomas, and. a… If we reverted that same thing, would it fix it, or would it just make it…
• 21:36:56 · Alex Martemyanov: No, it won't fix it. It would, uh, prevent the new users to get in. But if there is a feature flag for that flow, it should, uh… Make it.
• 21:37:13 · JŁ: But would it still be valuable to do so? So…
• 21:37:18 · Alex Martemyanov: Maybe. If there is a fish or flock, that's… So I don't know, I haven't produced the second path for the returning user.
• 21:37:32 · JŁ: Okay, would you be able to check that if there's a feature flag for that like, perhaps we could do it like 2 ways like we could disable the flag. To help. Like, new users not to get into the state, and we would, like, release the hotfix too. Uh, to fix the state for users that already are in the loop.
• 21:37:58 · Alex Martemyanov: No, the users that are already in the loop, they are… they cannot be…
• 21:38:02 · JŁ: I know, I know. So for them, we'll release the hotfix. I'm just talking about new users, like whether like…
• 21:38:03 · Alex Martemyanov: fixed, but…
• 21:38:07 · Tomasz Patrzykat: Я Эленто, Нью, Юзерс?
• 21:38:08 · JŁ: Whether changing like switching the flag would actually make it for for better for them, or.
• 21:38:43 · Fernando Bunn: Okay, uh, do you still need me for anything here? All good?
• 21:38:48 · Alex Martemyanov: Yes, there is Sync Auto Restore. And it's… Is it mentioned there?
• 21:38:58 · Fernando Bunn: What is the problem with sync?
• 21:38:58 · Alex Martemyanov: Report it. Hmm?
• 21:39:02 · Fernando Bunn: What is the problem with sync?
• 21:39:05 · Alex Martemyanov: The second path that restores the data and the sink was mentioned as a trigger for the issue. The fixed app now uses the sync server. For chat history, only one sync is actually on. Fix context. Accumulated AI charts deleted. And… Sync Auto Restore.
• 21:40:09 · Fernando Bunn: Sorry, I'm not pulling. What is the issue related with syncing?
• 21:40:13 · Alex Martemyanov: Sync restore, the sync restoration prompt, if we disable this feature flag, the sync restoration prompt should not appear. So, it means the user won't get through the sync restoration flow, and they won't get. the. Yes. Yes, but it's… But it's the common feature flag for all these encoders, not only for the onboarding.
• 21:40:45 · Tomasz Patrzykat: О?
• 21:40:46 · Alex Martemyanov: It does not disable sync, it disables automatic recovery of previously preserved sync account. Hide the "Let's pick up where we left off" button. And restore my stuff during onboarding. Remove the automatic restore path from sync settings. User must use QR code or recovery code or manual setup. So basically, this is automatic keychain recovery from keychain for reinstallers, and they have to… They have to sync up manually again. We can disable the whole flow, and it will mitigate this issue. But disabled synchro. They're restored with SYNCFLOW.
• 21:41:37 · Tomasz Patrzykat: Возраисков. Дизайн Фул Сынг?
• 21:41:41 · Alex Martemyanov: The loss of the restoration we are seeing from the keychain item for installers.
• 21:41:44 · Tomasz Patrzykat: Угу?
• 21:41:53 · Alex Martemyanov: So they will have to manually restore things. Or lose data, if… They have no access to the QR code anymore. At least until the next… Reinstall. That's not the case. So I'd say, uh, since… my opinion, that since it's not the really everyday use case scenario when… where you left off. The Duck or Yea chat. It may… it may be… And not a good user experience, but it's not a user breakage. And we should not remove, we should not lose user data.
• 21:42:46 · Fernando Bunn: Yeah, so. The the. Just make sure I understand here, like, we have two different problems, right? Like, one is this onboarding thing that keeps getting into a loop. This is the incident thing that we… like, you have the PR and it's done, right? Another one is this sync issue that I didn't understand very well, but basically, if you reinstall the app, there's still sync information there that it shouldn't be. Right? The second one.
• 21:43:08 · Alex Martemyanov: No.
• 21:43:09 · Fernando Bunn: Is it an incident as well? Is it something that needs to be part of this?
• 21:43:12 · Alex Martemyanov: No, it's, uh, it's the same, it's the same issue.
• 21:43:16 · Fernando Bunn: So it's related to onboarding as well?
• 21:43:19 · Alex Martemyanov: Yes, so when you have… so how would they use… how the new users are getting into the loop? Because normally, and the issue was there for a while already, it was half a year since it was merged, but we've started receiving this incident only recently, with this sink restoration. It's a reaction, because…
• 21:43:40 · Fernando Bunn: Okay, so it… Okay, it feels like the other one is way more… pressing time-related, like, for us to fix as soon as possible, and this one is not. I don't know if I'm wrong here.
• 21:43:51 · Tomasz Patrzykat: Депендских? Дат. С. Мои андрестанников Да Ищи? Инту, Дис? Он, ореваясь?
• 21:44:10 · Alex Martemyanov: Everybody who is doing this in restoration.
• 21:44:21 · Tomasz Patrzykat: Генджек?
• 21:44:23 · Fernando Bunn: So… Yeah, so to reproduce this, I need to have, like, sync enable, and then… Delete the app, install the app again.
• 21:44:34 · Alex Martemyanov: No, uh, yes. Yes.
• 21:44:39 · Fernando Bunn: Okay.
• 21:44:39 · Alex Martemyanov: Delete the app and reinstall again and restore from sync.
• 21:44:47 · JŁ: So it doesn't occur to completely new users to DuckDuckGo?
• 21:44:53 · Alex Martemyanov: Hm?
• 21:44:54 · JŁ: It doesn't even.
• 21:44:54 · Alex Martemyanov: Sure.
• 21:44:56 · JŁ: New users to DuckDuckGo who never had an app before on their phone. This is not gonna be triggered for them. You're saying?
• 21:45:04 · Alex Martemyanov: Uh, they shouldn't, in general. So it's only the case when you… you kill the app and they got this screen. But it's been there for a while already, and we haven't received a lot of feedback about it. Before. So it's the condition that you have… that you should have three, uh… 3 consequences, 3 cases, 3 events in a row. So, missing this… button then. Entering the Dakaria chat and. Restarting the app with Duck AI Chart active.
• 21:45:44 · JŁ: So okay, okay, I need to double check. So you're saying, if I if I have a raw simulator. I shouldn't be able to even reproduce the issue.
• 21:45:55 · Alex Martemyanov: Perfect.
• 21:45:57 · JŁ: Okay. I can check. Just to be double sure.
• 21:46:03 · Alex Martemyanov: So, in general, if you just press the… you've got this button. It won't be there. Yes. Yes. That's the case.
• 21:46:58 · Tomasz Patrzykat: Ол. Пробэмс?
• 21:47:00 · JŁ: I think, I think it fixes them. Uh, at least the one that… that incident reports were telling about right? Alex. And Fernando, like, you were checking it…
• 21:47:13 · Alex Martemyanov: Yes, who is, uh… who is getting to the screen? They will have to do the fire button thing again, but the next time it's fixed. It won't appear anymore.
• 21:47:26 · JŁ: Okay.
• 21:47:31 · Alex Martemyanov: And there is sync auto restore success excel. Use parameter source equals onboarding. And for users who only tapped restore my stuff. There is another pixel. Let me just check it. And…
• 21:48:07 · Tomasz Patrzykat: Okay, just to, like, double check, so the fix that was linked here, like, 20 minutes ago, the iOS fix.ai fire on board registration loop 7147. This fixes the root cause that was exposing this incident, correct?
• 21:48:24 · Alex Martemyanov: Yes.
• 21:48:25 · Tomasz Patrzykat: Where are we with releasing this?
• 21:48:28 · Alex Martemyanov: PR open against current internal orders.
• 21:48:35 · Tomasz Patrzykat: Okay, Joey, are you blocked, or do you need, like, any help with that, or just, like, try to engage more people if needed?
• 21:48:44 · JŁ: No, we will need Haniel, who is on the release rotation this week.
• 21:48:48 · Alex Martemyanov: Increase.
• 21:48:49 · Tomasz Patrzykat: Mm-hmm.
• 21:48:50 · JŁ: To to to the hot fix process for us. So basically.
• 21:48:56 · Tomasz Patrzykat: Okay.
• 21:48:57 · JŁ: Okay.
• 21:48:58 · Tomasz Patrzykat: Could you ping him in the channel?
• 21:48:58 · Tomasz Patrzykat: Включи, Пинхим, Индо, Инде, Ченнел?
• 21:48:59 · JŁ: Thank you. Hanyu, Hanyu was on the previous Zoom call. He just had to escape to some other meeting, but we can, like, someone… someone heard back, and…
• 21:49:06 · Tomasz Patrzykat: Okay?
• 21:49:11 · JŁ: and like the the there, there is a clear process and instruction how to how to do it. And so like, yeah, it should all be documented. So it should be fine, I I guess.
• 21:49:24 · Tomasz Patrzykat: So who can derive this, like releasing this?
• 21:49:27 · JŁ: Hanyu, I think. Like, that's the process, at least.
• 21:49:30 · Tomasz Patrzykat: Yeah, but he's not dead.
• 21:49:32 · JŁ: Uh, I will, I will summon her. Just, just…
• 21:49:35 · Tomasz Patrzykat: Okay, thank you.
• 21:50:02 · Fernando Bunn: Okay, I'm gonna…
• 21:50:04 · Alex Martemyanov: And for the…
• 21:50:04 · Fernando Bunn: Drop.
• 21:50:06 · Alex Martemyanov: For the prompt shown, I'm just looking at the board. And we've got, uh… 800 for the prompt shown. and hundred and sixty-nine daily. And… Onboarding restored. Dot. Strong and boldly. Restore Tap to Nick. Still… Okay. Okay. Oh, it's absorption. Yeah, here we go. 400 daily. On both Android and iPhone, unique links. Hi, Alice. iOS tablet and phone, 258 daily users going through the flow, supposedly, but not finishing. I think auto restore success. I'll do those. Click on to restore success. Success. I'm nervous. iOS phone 272, which kind of matches the… And… Pretty sure. State chapter. The source.
• 21:51:40 · Tomasz Patrzykat: Could you share the board or what are you looking for?
• 21:51:40 · Alex Martemyanov: Source. Uh, yes, one… one moment, any of these drinks? Next, uh… It's… Mm-hmm.
• 21:51:59 · JŁ: Honey, sorry, I brought you here. Quick context. So we have a fix.
• 21:52:07 · Alex Martemyanov: Thank you.
• 21:52:07 · JŁ: Um, it's been, like, reviewed.
• 21:52:09 · Hanyu Tang: Has it merged? No, it's not worked. Okay.
• 21:52:10 · JŁ: No, not yet. I think it's been reviewed by Fernando and Hassan. It's been proven to be working, and so we would be hot-fixing it.
• 21:52:23 · Hanyu Tang: Okay.
• 21:52:24 · JŁ: Since you're on the release rotation, could you be, like, would you be willing to do it? there's a process, there's a process, like, described in the documentation, I can share the task, like, even, like, here.
• 21:52:33 · Hanyu Tang: Yeah. Okay, sure.
• 21:52:39 · JŁ: And. to do it. Basically, it should be automated almost on every step. But yeah, like in in kind of case, like, feel free to like ping anyone for help.
• 21:52:53 · Hanyu Tang: Okay. Okay, yeah. Uh…
• 21:52:57 · Alex Martemyanov: Yeah, just…
• 21:52:58 · Hanyu Tang: Sorry, the PR is the fixed tax AI fire onboarding, right?
• 21:52:59 · Alex Martemyanov: Good question. Yeah, but it's against the…
• 21:53:03 · Hanyu Tang: It should be merged… It should be merged. It should not be merged into. 7241, right? It should be merged into me.
• 21:53:13 · Alex Martemyanov: Right.
• 21:53:14 · Hanyu Tang: Okay, yeah.
• 21:53:14 · JŁ: But…
• 21:53:14 · Alex Martemyanov: It should be merged to the previous one, and I'm just trying to figure out how to…
• 21:53:16 · Hanyu Tang: Yeah, yeah.
• 21:53:18 · JŁ: I think, basically, you need to start with the task, so, like, it tells you to, like, set up hotfix release branch from main, like, it happens all automatically, and then once it happens, once there's a branch.
• 21:53:19 · Alex Martemyanov: How to do this.
• 21:53:32 · JŁ: uh… done by… by CA… CI for you, you will basically, like, Alex will rebase his fix on top of that branch, which can be then merged, and… and, yeah, basically, then… And you can go step by step.
• 21:53:48 · Alex Martemyanov: Okay, so I'm waiting for the signal.
• 21:53:48 · Hanyu Tang: Okay, yeah, I already started the process.
• 21:53:51 · Alex Martemyanov: Okay, and the final question, so we have around 300 daily users, uh, getting to this state. Is it worth disabling the whole synchro restoration for those, or…
• 21:54:02 · Tomasz Patrzykat: Yeah, I think we should disable it, and then we'll, like, after hotfix, we'll just re-enable it. That should work, right? Because it should be then fixed.
• 21:54:12 · Alex Martemyanov: Yes, and there should be a button in settings, I guess.
• 21:54:16 · Tomasz Patrzykat: So who can disable this?
• 21:54:16 · Alex Martemyanov: Just in case. I can… Oh, pink, with a, with a. Where was the config? PR. In the chat, in the Metamorph later. I have to drop off for the mid-mortem now, and I'll do this. Disabling in parallel.
• 21:54:39 · Tomasz Patrzykat: Okay, so I can put you as a DRI for disabling the flow, right?
• 21:54:40 · Alex Martemyanov: Sounds good. Yep, yep.
• 21:54:46 · Tomasz Patrzykat: Okay, thank you.
• 21:54:47 · Alex Martemyanov: And waiting for a signal from Konya. Okay, thank you.