[2026-11-19] eagle: personal/projects/personal-os/personal-os-evidence/2026-11-19/follow-up-live-acceptance.md personal/projects/personal-os/personal-os-production-acceptance.md personal/projects/personal-os/personal-os-remediation-plan-2026-11-18.md work/zoom-transcripts/index.json work/zoom-transcripts/undated O-E optional kick-offs and postmortems nExkucRLSESP9SkgIBfCyg/transcript.json work/zoom-transcripts/undated O-E optional kick-offs and postmortems nExkucRLSESP9SkgIBfCyg/transcript.md

This commit is contained in:
Alexey Martemyanov committed 2026-11-19 22:49:19 +06:00
1 parent 7f38d9a364
commit b2dcf38779
6 files changed
+373

No files matched your search

@@ -369,3 +369,8 @@
- В рабочем журнале в 22:28 два чтения источников обзора завершились тайм-аутом около 31 секунды. В 22:32 нагрузка Mac выросла до 125,69, среди работающих процессов были Xcode и виртуальная машина. Это совпадение по времени, причина задержки каждого запроса отдельно не установлена. - В рабочем журнале в 22:28 два чтения источников обзора завершились тайм-аутом около 31 секунды. В 22:32 нагрузка Mac выросла до 125,69, среди работающих процессов были Xcode и виртуальная машина. Это совпадение по времени, причина задержки каждого запроса отдельно не установлена.
- До правки на живом портале в 1280 и 390 px заголовок «Главное на неделе» отсутствовал, пока задерживался обычный ответ обзора. После правки на том же пути отдельное чтение показало «1‑Must · Hack Phase - 2-3 days» и текущий Top Priority до получения полного ответа. Когда полный обзор завершился, оба окна сохранили один блок приоритетов. Asana подтвердила открытый этап `1220528695906856` и его родителя `1220371514136367`. Контроль исключил непроверенный профиль и отсутствующие карточки, совпавший полный ответ сохранил фокус на ссылке. [Неделя](http://127.0.0.1:4317/#/weekly-review) · [Этап Hack Phase](https://app.asana.com/1/137249556945/task/1220528695906856) - До правки на живом портале в 1280 и 390 px заголовок «Главное на неделе» отсутствовал, пока задерживался обычный ответ обзора. После правки на том же пути отдельное чтение показало «1‑Must · Hack Phase - 2-3 days» и текущий Top Priority до получения полного ответа. Когда полный обзор завершился, оба окна сохранили один блок приоритетов. Asana подтвердила открытый этап `1220528695906856` и его родителя `1220371514136367`. Контроль исключил непроверенный профиль и отсутствующие карточки, совпавший полный ответ сохранил фокус на ссылке. [Неделя](http://127.0.0.1:4317/#/weekly-review) · [Этап Hack Phase](https://app.asana.com/1/137249556945/task/1220528695906856)
## Родители в проверке полей Asana
- До правки рабочая проверка вернула 23 проверенные карточки и четыре дочерние карточки с неподтверждённым состоянием родителя. Прямая Asana подтвердила завершение трёх точных родителей `1207003732044068`, `1209153204826052` и `1203268333491284`. Четыре дочерние карточки сняты с ожидания сверки. Новые поручения по ним отсутствуют. Все 20 прежних замечаний совпали до и после.
- «Неделя → Входящие, проекты и карьерные цели → Проверка полей Asana» и «Мои задачи → Проверка полей Asana» проверены при 1280 и 390 px. Состояние родителей совпало в четырёх проходах. Контрольный случай отдельно подтвердил активного и приостановленного родителя, один запрос для общего родителя и сохранение неопределённости при отказе Asana. Все запросы к первичному источнику были на чтение. [Неделя](http://127.0.0.1:4317/#/weekly-review) · [Мои задачи](http://127.0.0.1:4317/#/my-tasks)
@@ -78,6 +78,7 @@
- **Транскрипты после встречи.** После слота Postmortem обход Zoom прочитал три раздела списка и обнаружил одну заметку без доступного транскрипта. Состояние системы показало этот пробел в двух размерах окна. Расписание Asana осталось доступно. [Доказательства](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md) - **Транскрипты после встречи.** После слота Postmortem обход Zoom прочитал три раздела списка и обнаружил одну заметку без доступного транскрипта. Состояние системы показало этот пробел в двух размерах окна. Расписание Asana осталось доступно. [Доказательства](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md)
- **Свежая проверка полей Asana.** Живая «Неделя» читает проверку при раскрытии и повторяет запрос после закрытия карточки. Пустой Project Advisor сверён с исходным проектом. Снятие замечания после добавления TD в реестр проверено на отдельном входе. [Доказательства](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md) - **Свежая проверка полей Asana.** Живая «Неделя» читает проверку при раскрытии и повторяет запрос после закрытия карточки. Пустой Project Advisor сверён с исходным проектом. Снятие замечания после добавления TD в реестр проверено на отдельном входе. [Доказательства](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md)
- **Приоритеты при долгой загрузке.** В двух размерах окна «Неделя» показала Hack Phase и Top Priority до получения полного обзора. Точные карточки и связь этапа с проектом сверены с Asana. [Доказательства](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md) - **Приоритеты при долгой загрузке.** В двух размерах окна «Неделя» показала Hack Phase и Top Priority до получения полного обзора. Точные карточки и связь этапа с проектом сверены с Asana. [Доказательства](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md)
- **Сверка родителя в Asana.** Три закрытых родителя подтвердили состояние четырёх ранее неопределённых карточек. В живой «Неделе» и «Моих задачах» остались те же 20 обоснованных находок. [Доказательства](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md)
- **План и приоритет.** E4 и недельный отбор принимают проверенную связь с действующим Top Priority и подтверждённые личные запросы. Production E5 сохранял текущий день, а рабочий браузер подтвердил совпадение действий, source run и ревизии дня/недели. Показанный объём действий не является постоянным лимитом рабочего дня. [Проверка приоритета](personal-os-evidence/2026-11-18/source-priority-validation.json) · [Проверка шести процессов](personal-os-evidence/2026-11-18/windmill-six-workflows.json) - **План и приоритет.** E4 и недельный отбор принимают проверенную связь с действующим Top Priority и подтверждённые личные запросы. Production E5 сохранял текущий день, а рабочий браузер подтвердил совпадение действий, source run и ревизии дня/недели. Показанный объём действий не является постоянным лимитом рабочего дня. [Проверка приоритета](personal-os-evidence/2026-11-18/source-priority-validation.json) · [Проверка шести процессов](personal-os-evidence/2026-11-18/windmill-six-workflows.json)
- **Asana.** Действует непрерывный worker с очередью и продолжением. Текущие назначения имеют преимущество перед завершённой историей. Частичный охват не объявляется полным; зависимости перепроверяются при допуске действия. [Зависимости](personal-os-evidence/2026-11-18/asana-dependencies-validation.json) - **Asana.** Действует непрерывный worker с очередью и продолжением. Текущие назначения имеют преимущество перед завершённой историей. Частичный охват не объявляется полным; зависимости перепроверяются при допуске действия. [Зависимости](personal-os-evidence/2026-11-18/asana-dependencies-validation.json)
@@ -509,3 +509,7 @@ Zoom используется для транскриптов состоявши
### D18-91 · Главный приоритет при задержке недельного обзора ### D18-91 · Главный приоритет при задержке недельного обзора
- Долгое чтение обсуждений и уточнений задерживало блок «Главное на неделе». При задержке портал отдельно читает приоритеты из сохранённого расчёта с подтверждённым профилем целей и сверяет точные карточки Asana. Когда приходит полный обзор, одинаковый блок не перестраивается. [Проверка](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md) - Долгое чтение обсуждений и уточнений задерживало блок «Главное на неделе». При задержке портал отдельно читает приоритеты из сохранённого расчёта с подтверждённым профилем целей и сверяет точные карточки Asana. Когда приходит полный обзор, одинаковый блок не перестраивается. [Проверка](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md)
### D18-92 · Прямая сверка состояния родителя
- В проверке полей четыре дочерние карточки долго ожидали свежего снимка завершённых родителей. При раскрытии портал сверяет неподтверждённого родителя непосредственно в Asana и применяет правило к прочитанному состоянию. При отказе источника остаётся запрос на сверку, прежнее состояние не принимается за текущее. [Проверка](personal-os-evidence/2026-11-19/follow-up-live-acceptance.md)
+12
View File
@@ -388,6 +388,18 @@
"folder": "2026-11-17 Apple Weekly Team Meeting - Zoom ID GJhjFoDDRY6PkqEUdM-YFg", "folder": "2026-11-17 Apple Weekly Team Meeting - Zoom ID GJhjFoDDRY6PkqEUdM-YFg",
"sha256": "cd06af7a61ede6d4cd5d78b6ed94d9c6650c99a66d637cdf0e2eba0fef6390c5", "sha256": "cd06af7a61ede6d4cd5d78b6ed94d9c6650c99a66d637cdf0e2eba0fef6390c5",
"cues": 48 "cues": 48
},
"nExkucRLSESP9SkgIBfCyg": {
"title": "O-E optional kick-offs and postmortems",
"meeting_date": null,
"meeting_date_source": "unknown",
"meeting_start_at": null,
"note_created_at": "2026-11-19T16:00:57.657Z",
"note_modified_at": "2026-11-19T16:00:57.657Z",
"source_url": "https://us01mynotes.zoom.us/nExkucRLSESP9SkgIBfCyg",
"folder": "undated O-E optional kick-offs and postmortems nExkucRLSESP9SkgIBfCyg",
"sha256": "ef8699da83e01294e7ff7c48b2f412a9b0d41e6efdc909bf3444e879e5a464b7",
"cues": 54
} }
} }
} }
@@ -0,0 +1,283 @@
{
"schema_version": 1,
"note_id": "nExkucRLSESP9SkgIBfCyg",
"title": "O-E optional kick-offs and postmortems",
"meeting_date": null,
"meeting_date_source": "unknown",
"meeting_start_at": null,
"note_created_at": "2026-11-19T16:00:57.657Z",
"note_modified_at": "2026-11-19T16:00:57.657Z",
"source_url": "https://us01mynotes.zoom.us/nExkucRLSESP9SkgIBfCyg",
"transcript": [
{
"time": "22:00:57",
"speaker": "Max Tsoy",
"text": "That will continue. So."
},
{
"time": "22:00:57",
"speaker": "russell holt",
"text": "Okay."
},
{
"time": "22:01:00",
"speaker": "Max Tsoy",
"text": "I think it's fine."
},
{
"time": "22:01:00",
"speaker": "russell holt",
"text": "Yeah. Okay."
},
{
"time": "22:01:07",
"speaker": "Alex Martemyanov",
"text": "Hey."
},
{
"time": "22:01:07",
"speaker": "Max Tsoy",
"text": "Hey, everyone."
},
{
"time": "22:01:09",
"speaker": "russell holt",
"text": "Okay. What's happening, Alex? Alright."
},
{
"time": "22:01:13",
"speaker": "Alex Martemyanov",
"text": "Uh, yeah, uh, here for the postmortem, for the… CPM breakage investigation. Um, sure. Should I start?"
},
{
"time": "22:01:25",
"speaker": "russell holt",
"text": "Yep, congrats on the thumbs up from… Gabriel."
},
{
"time": "22:01:30",
"speaker": "Alex Martemyanov",
"text": "Uh, yeah, uh, this is, uh, good, uh… Conclusion, I think. And. So basically, the root cause investigation project, basically find this, I don't know what. Reproduced and fixed. Nobody reproduced. So the project did partially achieve the success criteria. Meaning the breakage span, which we had no initial idea of how widespread. It could be, saving. some false positives and reporting adjust during the project. So the goal was set mid-mortem three, I guess. And to get permanent stack state below 1% of daily active users. And it's kind of partially achieved. At least for iOS, it seems below, according to the data we've got. On macOS, it's still slightly above, but I'm not strong on it. So it may be some false positives and it may be some genuine issue, it may be some race conditions. So, there was no single root cause identified during the project. There was several, several possible issues. Some of them were genuine bugs, like tab management, not. not completely done correctly, and those were fixed. And there were several conditions that were produced in the test environment, not on the real browser. Um, including some, uh, background, uh… Thank you. Background process crash, networking process crash, error during loading, or return from the background state. Uh, so some of them, uh… Seen have affected the. the issue positively, some of them did not, although the test stands need to produce the possible issue, but obviously those who are not in real condition. So. Um… Well, what went well, there were debug pixels discussed during kickoff to be added. Um… They were added before the, um… Steps taken to fix. Mostly, um, but… it has its other side, uh, the release cycles that affected, uh, the telemetry data, and I had to move blindly before the next release cycle and update the data, and, uh… repeated several times. Um, because, uh… The data was not clear. They are not. Um… But, uh, after… receiving the extended attributed data. There were. more of, um… This makes sense. checked and possible conditions. Um… tried. There were 3 midmortems, um, so the scope was, uh, extended from a couple of weeks to… actual around 12 weeks, which included some sick time, hard days, and so on. But, uh, overall, it was, uh, highly underestimated. But, uh, all the, uh, scope growth was, uh. reprioritized during the midmortems. And I hope that everybody stayed. informed on how the project proceeds and what's currently being done. Um… Also, there was some measurable effects. The numbers are estimates, so those are not actual… So the not actual, issues caught, but estimates according to the data. Like, we got, lower and, lower bound, higher bound. And there was… some… drop… Probably it wouldn't be it wouldn't be correct to say that we have dropped from 8 to less than 1%."
},
{
"time": "22:06:37",
"speaker": "Alex Martemyanov",
"text": "But, uh… I would give it couple of percent. Not for real cases, Joe. Okay. Then there was a cleanup done. All the test attribution is now missing and permanent pixel added. Uh, there was some issues, uh, in reporting, like false positive for the error page, which was also identified and fixed. And an early court issue of base URL. not working correctly on older macOS versions, and, we got it during the internal phase, and. It's mitigated through the privacy config, so it didn't reach the users. Yeah. In the end, there was a permanent measurement edit and a couple of follow-ups to investigate the JavaScript side. From what didn't go well, I can… I already highlighted the estimate that was adjusted several times. didn't you to um. Changes in how we collect the data, in how we evaluate."
},
{
"time": "22:07:59",
"speaker": "russell holt",
"text": "Yeah, but that's fine. That's what we expected to happen."
},
{
"time": "22:08:01",
"speaker": "Alex Martemyanov",
"text": "If hypothesis."
},
{
"time": "22:08:04",
"speaker": "russell holt",
"text": "You kept coming with midmortems for us to decide if we need to go further, and you had… Multiple… Causes to investigate. So I don't think that's a bad thing, just for what it'."
},
{
"time": "22:08:17",
"speaker": "Alex Martemyanov",
"text": "Yeah, I can't say it's a bad thing because it was reasonable. And due to the amount of work done and hypothesis tested and actual things found, I think it should be acceptable. It was affected by the feedback loop length, so we had a two week release train, and then we had another two weeks release train for the fixes that were supposed to target the issues highlighted by the data. So effectively it was four week iteration per fix. And… The metric is… the metric got changed, and uh… their… you know, there was actual pitfall in the metrics, uh, so we got no baseline, as I already told, so the numbers are estimate, and we cannot tell how much of, uh… We cannot compare the initial data to the final outcomes because the original data was not completely aligned with the actual features we've got. Maybe it was good to keep the original metric there until the project completes, just to have an idea of how the original numbers are affected. And what actual drop there was. But, uh, having the, uh, having the measurements split into groups, uh, with the extended attribution, we got some… Give you some numbers that can give the overall idea. Uh, there were some hypotheses tried, like, um… guesses that didn't work out. Some of them were reproduced. So actual issue when you, when you crash the ground process for, for an iframe that keeps reload and, it actually may. may get CPM stuck. for, uh, for the whole window, but, uh, it was not, uh, reproducing in the wild, so probably not a widespread issue, but, like, a synthetic one. So the fix, uh, was not, uh, was not applied. And those were cleaned up. Um, hmm. Yeah, still there is a largest group that's not covered, around 40%. There's still no viable hypothesis of why it's happening, of why it can happen. seemingly normal events are received for macOS and iOS, so it's higher on macOS. Maybe it's false positive for some of the conditions that are. That haven't been taken into account. But… Maybe something happens inside the… JavaScript side, and, I've just made a final effort for a follow-up data analysis to it and JavaScript lifecycle metrics. Um, frankly, it's 50-50. I don't expect, uh, that it's for sure we will give us an answer, but it's worth evaluating, and I… I have added follow-up for… Uh, December 4th, it's, uh… 3 weeks from now, I guess. Maybe something could be highlighted. So… Thank you. What, what could be taken to, uh, keep? to make projects like this go faster. Yeah, as I already told, if we have the changing metrics, the baseline should be kept, and that would give us some final numbers for analysis to understand the real effect. I'm gonna defix his head. Yeah, as already told, we defined the… exit point, like, a metric that is acceptable during the third mid-mortem, and it should have been defined earlier, probably. But since we had no real numbers, only the estimates. Um, but yeah, we could, uh, we should have discussed it during the kickoff that, uh, having the reports at around 8%, 1% would be… an acceptable criteria, so at least something to adjust later on. And…"
},
{
"time": "22:13:18",
"speaker": "Alex Martemyanov",
"text": "Yeah, they're estimate fixes and align with weekly release train. Uh, this is more of, uh… More of a formal process, uh, set up for… such kind of… fixed batches, uh… And they should have the rollback criterion by themselves, so either they're right."
},
{
"time": "22:13:41",
"speaker": "russell holt",
"text": "Did you do that at all? Or did you figure out that you needed to do that later in the project?"
},
{
"time": "22:13:47",
"speaker": "Alex Martemyanov",
"text": "Uh, there were, uh, the feature flag tasks, just, uh, regular, regular process."
},
{
"time": "22:13:51",
"speaker": "russell holt",
"text": "Right."
},
{
"time": "22:13:54",
"speaker": "Alex Martemyanov",
"text": "But there was no, um… schedule set up for this. Like, we evaluate data on such date, and, uh, if we got metrics, like, such and such, we remove it. If the metrics, uh, is, uh, acceptable, we keep it. Uh, so it was, like. meant, assumed, but not written in the tasks."
},
{
"time": "22:14:20",
"speaker": "russell holt",
"text": "Yeah, I think this is a good point thinking about."
},
{
"time": "22:14:20",
"speaker": "Alex Martemyanov",
"text": "But, uh… Would be good point to to."
},
{
"time": "22:14:24",
"speaker": "russell holt",
"text": "aligning with the releases. Yeah, that's a good, I like this one a lot. It's not easy."
},
{
"time": "22:14:33",
"speaker": "Alex Martemyanov",
"text": "Thank you. Yeah, um… Yeah, and there were a couple of issues with the OS version. incompatibilities like brushes we've got on x86. Systems on macOS 12. And three Office 26 versions with base URL not working correctly."
},
{
"time": "22:15:01",
"speaker": "russell holt",
"text": "Right."
},
{
"time": "22:15:01",
"speaker": "Alex Martemyanov",
"text": "And, uh, those should, uh… So those had their own postmortems. And we've discussed that we want to extend our… OS version matrix tests. For the remaining work items. Yeah, basically the pixel is added to the failure dashboard that Max had set up. After we start receiving the data, it should be validated that it's received. And… they… JavaScript. Uh, the JavaScript init metrics, uh, should be activated on November 30, after we had them in the release. We decided to go, um… Uh, in stages allowed, to avoid extra, um, extra load, or… this. And uh… intensive initialization part, not to make things worse. Then get the data and disable the flag immediately, just not to keep it under the pressure. And analyze the data we have after December 4th when we get enough of it."
},
{
"time": "22:16:25",
"speaker": "russell holt",
"text": "Okay. December the 4th, alright."
},
{
"time": "22:16:29",
"speaker": "Alex Martemyanov",
"text": "Yeah, I've got the tasks set to myself to… I'll just later on. There was this decision needed for surrogate migration project. So."
},
{
"time": "22:16:39",
"speaker": "russell holt",
"text": "Okay."
},
{
"time": "22:16:45",
"speaker": "Alex Martemyanov",
"text": "Probably, we are not blocked here, since there is no… No. No exact reasoning. And, uh. Obvious issues are currently fixed, so probably it's not. It's not blocked. And… Yeah, and when we receive the data, we can arrange another… another call. I'll give an update in the recording tasks, whether you want to have another project to fix the… issues discovered, whether it's fixable on Javascript side, or we need another round. Investigation, maybe. And there is some cleanup remaining. Just for the fixes merged last week, to keep an eye, to keep the feature flags just in case and remove for later on in the end of November. That's basically all that we've got for the project, yeah, and we've got a comment from Gabriel, um… It's not, uh… Not a strong signal, I guess. But… It's… Mmhm. It's a good one, I think."
},
{
"time": "22:18:10",
"speaker": "Max Tsoy",
"text": "Always good to keep, uh, Gabe happy. Uh… I just wanted to note that. I agree that this kind of projects, they don't perfectly fit to like project. template, because… Um, it's in there, like, fundamentally, you would… We don't know what the problem is. So we. Uh, and all things… I mean, I'm quite impressed by your persistence and how you kept, like, pushing and finding new things. Really appreciate it. Um… On the metrics, though, like, one thing I still don't quite understand is… Um, before this project, right, our best. Measurement was the, uh, this, um… Stuck signal in the breakage reports. So, these ones… Um… And… Oh, what I don't understand, like, do you think it's still actually… At least correlating with the actual. bugs, or… Let me share my screen. Or do you think it's… we shouldn't look at it anymore? So I'm talking about this… waiting states. This is iOS, and this is macOS. Just by looking at it, it looks like it went down… It's light and well. Quite actually twice. On both platforms. But I don't know if… because you've added your own pixels, and of course, this is also filtered by… it's only from the breakage reports. Um… But, yeah, I don't know if it's a signal. Still."
},
{
"time": "22:20:08",
"speaker": "Alex Martemyanov",
"text": "Again, it's hard to tell here because even with the last effort of making the permanent pixels, we still got no exact matching for. the same URL tabs, for example, and I remember this is one of the follow-ups for me to do it to the tech design, so there is a way to match. the extension to the tabs exactly. That may be one of the issues. Uh, the error page, maybe another one, since, uh… Some of the breakage report. must be sent from the error page, I guess, which naturally won't receive these signals. And it may affect the breakage report metrics. Um… Some of the bugs were actually fixed, like, uh, especially for the long… for longer running browsers, when you rearrange tabs or close some windows, uh, the collected effects may… Mmhm. May actually affect the report numbers and the loading speed. Basically… Uh, the base URL, the update on… the update on launch, if we've got some wait time that originally were reported as, um… Uh, the web extension failure, but sometimes web extension used to load, but after 30 seconds, for example. The current metric, the permanent one that we've just nourished, should. should address most of them, I guess, and should be the most correct of all. It still does not match the tabs with the same URL exactly. But it makes its best effort. And. We'll see. In a couple of weeks. Indeed. Like, the real best effort numbers. And we still have no…"
},
{
"time": "22:22:16",
"speaker": "Max Tsoy",
"text": "So, the top, the top panels on that, on that dashboard, they, uh, we should look at them. First, right?"
},
{
"time": "22:22:22",
"speaker": "Alex Martemyanov",
"text": "Yeah, yeah, I hope so."
},
{
"time": "22:22:24",
"speaker": "Max Tsoy",
"text": "Okay. Uh, okay."
},
{
"time": "22:22:27",
"speaker": "Alex Martemyanov",
"text": "So again, we have no exact… breakage event. That's the main issue here. We try to speculate. When the page is loaded, and when there is enough time passed, and we had no signal, whether it's hang, or whether it's just slow response."
},
{
"time": "22:22:42",
"speaker": "Max Tsoy",
"text": "Mm-hmm. Yeah, yeah, it's approximatory, but as long as it, like, moves with, uh, with the, like, general, maybe absolute numbers, or…"
},
{
"time": "22:22:48",
"speaker": "Alex Martemyanov",
"text": "Are you sure?"
},
{
"time": "22:22:58",
"speaker": "Max Tsoy",
"text": "Not useful, but the changes are. Um, and yeah, so the… I think it's, like, a side effect, but they… Uh, what you found, you found a way. to map the, uh, internal tab IDs, like extension tab IDs, to the native tab IDs."
},
{
"time": "22:23:21",
"speaker": "Alex Martemyanov",
"text": "Yep."
},
{
"time": "22:23:22",
"speaker": "Max Tsoy",
"text": "As I understand, and this is… I think this is also a great find. I mean, it's not… it's unrelated to the… Kind of, uh, to the goals of this project, but… It's one of the two biggest… downsides that we've… that we had with the extension, and the second is what Sam just found. is how to push data from native to the extension. Basically, I think Sam found a way to wake up."
},
{
"time": "22:23:49",
"speaker": "Alex Martemyanov",
"text": "Mm-hmm."
},
{
"time": "22:23:53",
"speaker": "Max Tsoy",
"text": "forcibly wake up the service worker from the native side."
},
{
"time": "22:23:55",
"speaker": "Alex Martemyanov",
"text": "Right."
},
{
"time": "22:23:57",
"speaker": "Max Tsoy",
"text": "Um…"
},
{
"time": "22:23:57",
"speaker": "Alex Martemyanov",
"text": "That's great. Please ping me on it because I failed to find it. I tried as well, but I didn't. Yeah. So, yeah, I'll update the tech design, uh… Hopefully tomorrow. And so basically, it's just an easy mapping. I'll just post a comment. If it does for me, for myself. Okay, thanks."
},
{
"time": "22:24:28",
"speaker": "russell holt",
"text": "Okay. Alex, just FYI, I'm going to put a monitoring task in O-E. Made a project and linked to your. Follow up. It's in the description and assign it to you. Just as a way to track them."
},
{
"time": "22:24:46",
"speaker": "Alex Martemyanov",
"text": "Okay."
},
{
"time": "22:24:54",
"speaker": "russell holt",
"text": "What are we doing? Okay, thank you. Awesome, Alex. Good work. Chris, I think you're up."
},
{
"time": "22:25:02",
"speaker": "Alex Martemyanov",
"text": "Yes."
},
{
"time": "22:25:03",
"speaker": "Max Tsoy",
"text": "Thanks, Sarah. Thanks, Nicole."
},
{
"time": "22:25:05",
"speaker": "CB",
"text": "Uh, yes."
},
{
"time": "22:25:06",
"speaker": "Alex Martemyanov",
"text": "Thanks."
},
{
"time": "22:25:11",
"speaker": "CB",
"text": "Okay, just, uh, pinged, uh, Domin, and he said he's joining, so… Give us one second."
}
]
}
@@ -0,0 +1,68 @@
---
title: "O-E optional kick-offs and postmortems"
date: null
date_source: unknown
zoom_note_id: "nExkucRLSESP9SkgIBfCyg"
source: "https://us01mynotes.zoom.us/nExkucRLSESP9SkgIBfCyg"
---
# O-E optional kick-offs and postmortems
[Zoom My Notes](https://us01mynotes.zoom.us/nExkucRLSESP9SkgIBfCyg)
## Transcript
- **22:00:57 · Max Tsoy:** That will continue. So.
- **22:00:57 · russell holt:** Okay.
- **22:01:00 · Max Tsoy:** I think it's fine.
- **22:01:00 · russell holt:** Yeah. Okay.
- **22:01:07 · Alex Martemyanov:** Hey.
- **22:01:07 · Max Tsoy:** Hey, everyone.
- **22:01:09 · russell holt:** Okay. What's happening, Alex? Alright.
- **22:01:13 · Alex Martemyanov:** Uh, yeah, uh, here for the postmortem, for the… CPM breakage investigation. Um, sure. Should I start?
- **22:01:25 · russell holt:** Yep, congrats on the thumbs up from… Gabriel.
- **22:01:30 · Alex Martemyanov:** Uh, yeah, uh, this is, uh, good, uh… Conclusion, I think. And. So basically, the root cause investigation project, basically find this, I don't know what. Reproduced and fixed. Nobody reproduced. So the project did partially achieve the success criteria. Meaning the breakage span, which we had no initial idea of how widespread. It could be, saving. some false positives and reporting adjust during the project. So the goal was set mid-mortem three, I guess. And to get permanent stack state below 1% of daily active users. And it's kind of partially achieved. At least for iOS, it seems below, according to the data we've got. On macOS, it's still slightly above, but I'm not strong on it. So it may be some false positives and it may be some genuine issue, it may be some race conditions. So, there was no single root cause identified during the project. There was several, several possible issues. Some of them were genuine bugs, like tab management, not. not completely done correctly, and those were fixed. And there were several conditions that were produced in the test environment, not on the real browser. Um, including some, uh, background, uh… Thank you. Background process crash, networking process crash, error during loading, or return from the background state. Uh, so some of them, uh… Seen have affected the. the issue positively, some of them did not, although the test stands need to produce the possible issue, but obviously those who are not in real condition. So. Um… Well, what went well, there were debug pixels discussed during kickoff to be added. Um… They were added before the, um… Steps taken to fix. Mostly, um, but… it has its other side, uh, the release cycles that affected, uh, the telemetry data, and I had to move blindly before the next release cycle and update the data, and, uh… repeated several times. Um, because, uh… The data was not clear. They are not. Um… But, uh, after… receiving the extended attributed data. There were. more of, um… This makes sense. checked and possible conditions. Um… tried. There were 3 midmortems, um, so the scope was, uh, extended from a couple of weeks to… actual around 12 weeks, which included some sick time, hard days, and so on. But, uh, overall, it was, uh, highly underestimated. But, uh, all the, uh, scope growth was, uh. reprioritized during the midmortems. And I hope that everybody stayed. informed on how the project proceeds and what's currently being done. Um… Also, there was some measurable effects. The numbers are estimates, so those are not actual… So the not actual, issues caught, but estimates according to the data. Like, we got, lower and, lower bound, higher bound. And there was… some… drop… Probably it wouldn't be it wouldn't be correct to say that we have dropped from 8 to less than 1%.
- **22:06:37 · Alex Martemyanov:** But, uh… I would give it couple of percent. Not for real cases, Joe. Okay. Then there was a cleanup done. All the test attribution is now missing and permanent pixel added. Uh, there was some issues, uh, in reporting, like false positive for the error page, which was also identified and fixed. And an early court issue of base URL. not working correctly on older macOS versions, and, we got it during the internal phase, and. It's mitigated through the privacy config, so it didn't reach the users. Yeah. In the end, there was a permanent measurement edit and a couple of follow-ups to investigate the JavaScript side. From what didn't go well, I can… I already highlighted the estimate that was adjusted several times. didn't you to um. Changes in how we collect the data, in how we evaluate.
- **22:07:59 · russell holt:** Yeah, but that's fine. That's what we expected to happen.
- **22:08:01 · Alex Martemyanov:** If hypothesis.
- **22:08:04 · russell holt:** You kept coming with midmortems for us to decide if we need to go further, and you had… Multiple… Causes to investigate. So I don't think that's a bad thing, just for what it'.
- **22:08:17 · Alex Martemyanov:** Yeah, I can't say it's a bad thing because it was reasonable. And due to the amount of work done and hypothesis tested and actual things found, I think it should be acceptable. It was affected by the feedback loop length, so we had a two week release train, and then we had another two weeks release train for the fixes that were supposed to target the issues highlighted by the data. So effectively it was four week iteration per fix. And… The metric is… the metric got changed, and uh… their… you know, there was actual pitfall in the metrics, uh, so we got no baseline, as I already told, so the numbers are estimate, and we cannot tell how much of, uh… We cannot compare the initial data to the final outcomes because the original data was not completely aligned with the actual features we've got. Maybe it was good to keep the original metric there until the project completes, just to have an idea of how the original numbers are affected. And what actual drop there was. But, uh, having the, uh, having the measurements split into groups, uh, with the extended attribution, we got some… Give you some numbers that can give the overall idea. Uh, there were some hypotheses tried, like, um… guesses that didn't work out. Some of them were reproduced. So actual issue when you, when you crash the ground process for, for an iframe that keeps reload and, it actually may. may get CPM stuck. for, uh, for the whole window, but, uh, it was not, uh, reproducing in the wild, so probably not a widespread issue, but, like, a synthetic one. So the fix, uh, was not, uh, was not applied. And those were cleaned up. Um, hmm. Yeah, still there is a largest group that's not covered, around 40%. There's still no viable hypothesis of why it's happening, of why it can happen. seemingly normal events are received for macOS and iOS, so it's higher on macOS. Maybe it's false positive for some of the conditions that are. That haven't been taken into account. But… Maybe something happens inside the… JavaScript side, and, I've just made a final effort for a follow-up data analysis to it and JavaScript lifecycle metrics. Um, frankly, it's 50-50. I don't expect, uh, that it's for sure we will give us an answer, but it's worth evaluating, and I… I have added follow-up for… Uh, December 4th, it's, uh… 3 weeks from now, I guess. Maybe something could be highlighted. So… Thank you. What, what could be taken to, uh, keep? to make projects like this go faster. Yeah, as I already told, if we have the changing metrics, the baseline should be kept, and that would give us some final numbers for analysis to understand the real effect. I'm gonna defix his head. Yeah, as already told, we defined the… exit point, like, a metric that is acceptable during the third mid-mortem, and it should have been defined earlier, probably. But since we had no real numbers, only the estimates. Um, but yeah, we could, uh, we should have discussed it during the kickoff that, uh, having the reports at around 8%, 1% would be… an acceptable criteria, so at least something to adjust later on. And…
- **22:13:18 · Alex Martemyanov:** Yeah, they're estimate fixes and align with weekly release train. Uh, this is more of, uh… More of a formal process, uh, set up for… such kind of… fixed batches, uh… And they should have the rollback criterion by themselves, so either they're right.
- **22:13:41 · russell holt:** Did you do that at all? Or did you figure out that you needed to do that later in the project?
- **22:13:47 · Alex Martemyanov:** Uh, there were, uh, the feature flag tasks, just, uh, regular, regular process.
- **22:13:51 · russell holt:** Right.
- **22:13:54 · Alex Martemyanov:** But there was no, um… schedule set up for this. Like, we evaluate data on such date, and, uh, if we got metrics, like, such and such, we remove it. If the metrics, uh, is, uh, acceptable, we keep it. Uh, so it was, like. meant, assumed, but not written in the tasks.
- **22:14:20 · russell holt:** Yeah, I think this is a good point thinking about.
- **22:14:20 · Alex Martemyanov:** But, uh… Would be good point to to.
- **22:14:24 · russell holt:** aligning with the releases. Yeah, that's a good, I like this one a lot. It's not easy.
- **22:14:33 · Alex Martemyanov:** Thank you. Yeah, um… Yeah, and there were a couple of issues with the OS version. incompatibilities like brushes we've got on x86. Systems on macOS 12. And three Office 26 versions with base URL not working correctly.
- **22:15:01 · russell holt:** Right.
- **22:15:01 · Alex Martemyanov:** And, uh, those should, uh… So those had their own postmortems. And we've discussed that we want to extend our… OS version matrix tests. For the remaining work items. Yeah, basically the pixel is added to the failure dashboard that Max had set up. After we start receiving the data, it should be validated that it's received. And… they… JavaScript. Uh, the JavaScript init metrics, uh, should be activated on November 30, after we had them in the release. We decided to go, um… Uh, in stages allowed, to avoid extra, um, extra load, or… this. And uh… intensive initialization part, not to make things worse. Then get the data and disable the flag immediately, just not to keep it under the pressure. And analyze the data we have after December 4th when we get enough of it.
- **22:16:25 · russell holt:** Okay. December the 4th, alright.
- **22:16:29 · Alex Martemyanov:** Yeah, I've got the tasks set to myself to… I'll just later on. There was this decision needed for surrogate migration project. So.
- **22:16:39 · russell holt:** Okay.
- **22:16:45 · Alex Martemyanov:** Probably, we are not blocked here, since there is no… No. No exact reasoning. And, uh. Obvious issues are currently fixed, so probably it's not. It's not blocked. And… Yeah, and when we receive the data, we can arrange another… another call. I'll give an update in the recording tasks, whether you want to have another project to fix the… issues discovered, whether it's fixable on Javascript side, or we need another round. Investigation, maybe. And there is some cleanup remaining. Just for the fixes merged last week, to keep an eye, to keep the feature flags just in case and remove for later on in the end of November. That's basically all that we've got for the project, yeah, and we've got a comment from Gabriel, um… It's not, uh… Not a strong signal, I guess. But… It's… Mmhm. It's a good one, I think.
- **22:18:10 · Max Tsoy:** Always good to keep, uh, Gabe happy. Uh… I just wanted to note that. I agree that this kind of projects, they don't perfectly fit to like project. template, because… Um, it's in there, like, fundamentally, you would… We don't know what the problem is. So we. Uh, and all things… I mean, I'm quite impressed by your persistence and how you kept, like, pushing and finding new things. Really appreciate it. Um… On the metrics, though, like, one thing I still don't quite understand is… Um, before this project, right, our best. Measurement was the, uh, this, um… Stuck signal in the breakage reports. So, these ones… Um… And… Oh, what I don't understand, like, do you think it's still actually… At least correlating with the actual. bugs, or… Let me share my screen. Or do you think it's… we shouldn't look at it anymore? So I'm talking about this… waiting states. This is iOS, and this is macOS. Just by looking at it, it looks like it went down… It's light and well. Quite actually twice. On both platforms. But I don't know if… because you've added your own pixels, and of course, this is also filtered by… it's only from the breakage reports. Um… But, yeah, I don't know if it's a signal. Still.
- **22:20:08 · Alex Martemyanov:** Again, it's hard to tell here because even with the last effort of making the permanent pixels, we still got no exact matching for. the same URL tabs, for example, and I remember this is one of the follow-ups for me to do it to the tech design, so there is a way to match. the extension to the tabs exactly. That may be one of the issues. Uh, the error page, maybe another one, since, uh… Some of the breakage report. must be sent from the error page, I guess, which naturally won't receive these signals. And it may affect the breakage report metrics. Um… Some of the bugs were actually fixed, like, uh, especially for the long… for longer running browsers, when you rearrange tabs or close some windows, uh, the collected effects may… Mmhm. May actually affect the report numbers and the loading speed. Basically… Uh, the base URL, the update on… the update on launch, if we've got some wait time that originally were reported as, um… Uh, the web extension failure, but sometimes web extension used to load, but after 30 seconds, for example. The current metric, the permanent one that we've just nourished, should. should address most of them, I guess, and should be the most correct of all. It still does not match the tabs with the same URL exactly. But it makes its best effort. And. We'll see. In a couple of weeks. Indeed. Like, the real best effort numbers. And we still have no…
- **22:22:16 · Max Tsoy:** So, the top, the top panels on that, on that dashboard, they, uh, we should look at them. First, right?
- **22:22:22 · Alex Martemyanov:** Yeah, yeah, I hope so.
- **22:22:24 · Max Tsoy:** Okay. Uh, okay.
- **22:22:27 · Alex Martemyanov:** So again, we have no exact… breakage event. That's the main issue here. We try to speculate. When the page is loaded, and when there is enough time passed, and we had no signal, whether it's hang, or whether it's just slow response.
- **22:22:42 · Max Tsoy:** Mm-hmm. Yeah, yeah, it's approximatory, but as long as it, like, moves with, uh, with the, like, general, maybe absolute numbers, or…
- **22:22:48 · Alex Martemyanov:** Are you sure?
- **22:22:58 · Max Tsoy:** Not useful, but the changes are. Um, and yeah, so the… I think it's, like, a side effect, but they… Uh, what you found, you found a way. to map the, uh, internal tab IDs, like extension tab IDs, to the native tab IDs.
- **22:23:21 · Alex Martemyanov:** Yep.
- **22:23:22 · Max Tsoy:** As I understand, and this is… I think this is also a great find. I mean, it's not… it's unrelated to the… Kind of, uh, to the goals of this project, but… It's one of the two biggest… downsides that we've… that we had with the extension, and the second is what Sam just found. is how to push data from native to the extension. Basically, I think Sam found a way to wake up.
- **22:23:49 · Alex Martemyanov:** Mm-hmm.
- **22:23:53 · Max Tsoy:** forcibly wake up the service worker from the native side.
- **22:23:55 · Alex Martemyanov:** Right.
- **22:23:57 · Max Tsoy:** Um…
- **22:23:57 · Alex Martemyanov:** That's great. Please ping me on it because I failed to find it. I tried as well, but I didn't. Yeah. So, yeah, I'll update the tech design, uh… Hopefully tomorrow. And so basically, it's just an easy mapping. I'll just post a comment. If it does for me, for myself. Okay, thanks.
- **22:24:28 · russell holt:** Okay. Alex, just FYI, I'm going to put a monitoring task in O-E. Made a project and linked to your. Follow up. It's in the description and assign it to you. Just as a way to track them.
- **22:24:46 · Alex Martemyanov:** Okay.
- **22:24:54 · russell holt:** What are we doing? Okay, thank you. Awesome, Alex. Good work. Chris, I think you're up.
- **22:25:02 · Alex Martemyanov:** Yes.
- **22:25:03 · Max Tsoy:** Thanks, Sarah. Thanks, Nicole.
- **22:25:05 · CB:** Uh, yes.
- **22:25:06 · Alex Martemyanov:** Thanks.
- **22:25:11 · CB:** Okay, just, uh, pinged, uh, Domin, and he said he's joining, so… Give us one second.