hmn/src/templates/src/fishbowls/testing/testing.html

<div class="chatlog">
<!-- Emojis by Twitter's Twemoji https://twemoji.twitter.com/ -->

<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112063022429646898" class="chatlog__message-container" data-message-id="1112063022429646898"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112063022429646898">May 27, 2023 10:01 AM</a></span></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Hey everybody, welcome to another fishbowl!</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="fishbowl"><img class="chatlog__emoji chatlog__emoji--small" alt="fishbowl" src="files/707755812277321760-F2EB5.png" loading="lazy"> <span class="chatlog__reaction-count">8</span></div></div></div></div></div></div><div id="chatlog__message-container-1112064410152874135" class="chatlog__message-container" data-message-id="1112064410152874135"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:06 AM">10:06</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The topic for today is testing. Anyone who has ever worked a programming job knows that testing is Important, but there's a lot of disagreement in industry about how to test. Within this community, I think there is often a sentiment against testing, or at least that the types of testing most people do are ineffective.</span></div></div></div></div><div id="chatlog__message-container-1112064712478306415" class="chatlog__message-container" data-message-id="1112064712478306415"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:08 AM">10:08</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">But despite this I think we all know that some kind of testing is in fact important. So the goal of this fishbowl is to examine our experiences with testing, and to try to learn how to think about testing and how to write effective tests.</span></div></div></div></div><div id="chatlog__message-container-1112065084521467986" class="chatlog__message-container" data-message-id="1112065084521467986"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:09 AM">10:09</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Our participants today are:
- Kartik Agaram (<span class="chatlog__markdown-mention" title="Kartik Agaram#9408">@Kartik Agaram</span>)
- Phil Homan (<span class="chatlog__markdown-mention" title="Phil H#5516">@Phil H</span>)
- Andrew Reece (<span class="chatlog__markdown-mention" title="azmr#4870">@Andrew (azmr)</span>)
- Demetri Spanos (<span class="chatlog__markdown-mention" title="demetrispanos#5588">@demetrispanos</span>)
- Phillip Trudeau (<span class="chatlog__markdown-mention" title="philliptrudeau#8865">@Phillip Trudeau</span>)
- and myself, as moderator
Side conversation is in <span class="chatlog__markdown-mention">#fishbowl-audience</span> starting here: <a href="https://discord.com/channels/239737791225790464/708458209131757598/1112064639564517457" onclick="scrollToMessage(event, '1112064639564517457')">https://discord.com/channels/239737791225790464/708458209131757598/1112064639564517457</a></span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:19 AM">(edited)</span></div></div></div></div><div id="chatlog__message-container-1112065225437483158" class="chatlog__message-container" data-message-id="1112065225437483158"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:10 AM">10:10</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">To start, I'd love if everyone could introduce themselves and talk briefly about their experiences with testing in the past!</span></div></div></div></div><div id="chatlog__message-container-1112065245402378310" class="chatlog__message-container" data-message-id="1112065245402378310"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:10 AM">10:10</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I'll start...</span></div></div></div></div><div id="chatlog__message-container-1112065625225953300" class="chatlog__message-container" data-message-id="1112065625225953300"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:11 AM">10:11</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I'm Ben, community lead, and most of my experience with testing was at a web development company for the past several years. We had automated tests for both our frontend and backend, but mostly for our frontend for some reason. We also had manual QA and large-scale automated end-to-end tests we would run before shipping. EDIT: As of recently I'm now at Mozilla on the WebAssembly team, and am coming up to speed on our test practices for Firefox and SpiderMonkey in particular.</span> <span class="chatlog__edited-timestamp" title="05/27/2023 12:18 PM">(edited)</span></div></div></div></div><div id="chatlog__message-container-1112065680020357160" class="chatlog__message-container" data-message-id="1112065680020357160"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:11 AM">10:11</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The frontend testing in particular was <em>really bad</em> in my opinion, and we might have opportunity to talk about that later...</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112066114432811108" class="chatlog__message-container" data-message-id="1112066114432811108"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112066114432811108">May 27, 2023 10:13 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Good morning, I'm Phil. I will mainly be talking about testing from the perspective of a live service engineer. I worked at Microsoft for ~3.5 yrs on some of the core web services for O365. Most of the testing was focused on live monitoring of the health of the deployed services, verifying the functionality of new features, and constantly improving our processes with weekly reviews. I currently work at a small indie game studio (Pocketwatch Games) in the earlier stages of development where writing tests is not part of my day to day work.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112067172240474123" class="chatlog__message-container" data-message-id="1112067172240474123"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112067172240474123">May 27, 2023 10:17 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I'm a computational mathematician working mainly in AI/ML, but also sometimes drones, robots, and sensing sytems. I've also worked alongside web/mobile programmers for many years, so I have some osmotic knowledge. 

Broadly, I consider testing to be part of a spectrum of "static analysis" tools. I had a brief phase where I took the popular unit-test/TDD style for a trial and decided it definitely was not for me (my work often has problems without clear correct answers).

Nowadays I evaluate testing in terms of likely programmer time saved. Usually that means integration/soak tests, and rarely fine-grain testing like unit tests.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112067703147085884" class="chatlog__message-container" data-message-id="1112067703147085884"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112067703147085884">May 27, 2023 10:20 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Hey all, I'm Andrew. I'm the lead dev on WhiteBox (a tool for showing how your code behaves as you write it / a live debugger / a runtime data timeline).

Our codebase makes liberal use of statically defined assumptions &amp; (custom) runtime asserts. Some of the more "infrastructure" code has unit tests, but we primarily rely on "integration tests" for testing user-facing behaviour.

The WhiteBox team spent some time looking into fuzz-testing &amp; property testing, which both look promising, but they dropped in priority and we never fully implemented either.

We also have a couple of ideas about how the tool itself could improve testing, primarily around where testing fits in the workflow, and how to save/generate complex data contexts to test against as an alternative to mocking.</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:23 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112068117519159376" class="chatlog__message-container" data-message-id="1112068117519159376"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112068117519159376">May 27, 2023 10:21 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Hi, I'm Phillip, I'm a game dev. I co-wrote a multiplayer physics game with Miles ( <span class="chatlog__markdown-mention" title="notnullnotvoid#6524">@notnullnotvoid</span> ) where part of the problem statement was to implement low-latency rollback netcode for Box2D's chaotic rigidbody dynamics (!!). We employed a scattershot of different Ways To Find Bugs, which from my perspective was what the testing I did was mainly about for that project. Last year, I joined Andrew working on WhiteBox, where I've actually been contributing to the testing system recently - which as you can see is much more systematic!</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:22 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112068755686690856" class="chatlog__message-container" data-message-id="1112068755686690856"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112068755686690856">May 27, 2023 10:24 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I've been all over the place in my life:
* pre-test-enlightenment: in academia working with microprocessor simulators, starting to build little CGI web pages
* test-enlightenment, moving to industry and building web apps, strong TDD
* gravitating towards the backend, offline jobs (decent number of tests, but they hindered as much as helped)
* going extremely low level (to design for testability from the ground up rather than wrap layers for things like browser testing)
* more graphical projects in the past year (certain aspects extremely hard to test)

I tend to be pretty pro tests. In my experience tests work really really well at small scales. The issues with tests tend to occur in large teams, and my lesson from that observation is that we just don't know how to grow teams without introducing entropy and dysfunction. It doesn't make sense to blame testing for that.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112069058783879242" class="chatlog__message-container" data-message-id="1112069058783879242"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112069058783879242">May 27, 2023 10:25 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Let's expand on that a bit - how effective have you all found different kinds of tests to be?</span></div></div></div></div><div id="chatlog__message-container-1112069469083271228" class="chatlog__message-container" data-message-id="1112069469083271228"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:27 AM">10:27</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">For me, for example, I often find benefit in writing tests for small, frequently-reused code like utility functions and data structures, since it helps flesh out edge cases. And I often find larger "integration" tests useful as sanity checks on the system. But a lot of the testing I did at my last job was neither, and I swear we had like a 95%+ false positive rate.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112069584141439016" class="chatlog__message-container" data-message-id="1112069584141439016"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112069584141439016">May 27, 2023 10:27 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I think it'd be worth carving out the categories real quick to start!</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up_2"><img class="chatlog__emoji chatlog__emoji--small" alt="👆" src="files/1f446-9CC34.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112069690727075911" class="chatlog__message-container" data-message-id="1112069690727075911"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112069690727075911">May 27, 2023 10:27 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">It is probably worth addressing a couple broad categories of testing, although I think people obsess over the boundaries more than they should</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112069733806768209" class="chatlog__message-container" data-message-id="1112069733806768209"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112069733806768209">May 27, 2023 10:28 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I look at it as the following costs vs benefits
costs
- writing the test
- writing the code so that it is testable
- creating paraphernalia (mocks etc.)
- running the test (every build?)
benefits
- verifying alleged invariants so you can detect breaking changes at build instead of run time</span></div></div></div></div><div id="chatlog__message-container-1112069896751304787" class="chatlog__message-container" data-message-id="1112069896751304787"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:28 AM">10:28</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">if I can write a test easily, and don't need to change my code to do so, and there is no paraphernalia, and the test runs instantly, and it prevents me from wasting hours in the future, then it's worthwhile</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112070129686171682" class="chatlog__message-container" data-message-id="1112070129686171682"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112070129686171682">May 27, 2023 10:29 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I think it's possibly worth splitting effectiveness by when the tests are written/triggered:
- before/while/immediately after writing some section of code: helping to hit all key cases while it's the primary focus
- when changing something else and unintentionally breaking old features (avoiding regressions)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112070544163098694" class="chatlog__message-container" data-message-id="1112070544163098694"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112070544163098694">May 27, 2023 10:31 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">For the cases where I've written unit tests, I've found them far more useful for the former (as an aid to writing code correctly). Integration tests seem more useful for the latter, as there are more unexpected interactions that break behaviour.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="point_up_2"><img class="chatlog__emoji chatlog__emoji--small" alt="👆" src="files/1f446-9CC34.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div><div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112070862603042847" class="chatlog__message-container" data-message-id="1112070862603042847"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112070862603042847">May 27, 2023 10:32 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">also relevant are the times that tests aren't worthwhile...</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112070304185979001" class="chatlog__message-container" data-message-id="1112070304185979001"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112070304185979001">May 27, 2023 10:30 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">for automated testing, I tend to think of "unit" and "integration" as the only two broad categories, where "unit" means isolated testing of a small piece of code (e.g. a function or data structure), and "integration" means testing the actual system end-to-end</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:30 AM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112070782286303364" class="chatlog__message-container" data-message-id="1112070782286303364"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112070782286303364">May 27, 2023 10:32 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">- unit testing - making sure a function/class/tiny subsystem/etc. doesn't have edge cases or doesn't crash - for my work on Happenlance, I did this rarely in special cases where it was easy to perform.
- Integration testing: Stressing major subsystems, either to proactively ensure no regressions or just to reactively catch bugs - I did this in a major way for Happenlance, and I should have done it more!
- System testing: "Really Running" the product and reactively catching bugs - I started to lean more heavily in my preferences toward this end over the course of development and post-release.</span></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112070877815771326" class="chatlog__message-container" data-message-id="1112070877815771326"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112069690727075911')"> It is probably worth addressing a couple broad categories of testing, although I think people obsess over the boundaries more than they should </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112070877815771326">May 27, 2023 10:32 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The categories of tests for me are really indicative of code in general. If your project is small, things that would be integration tests for others become unit tests for you. Life is good.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112071045436948650" class="chatlog__message-container" data-message-id="1112071045436948650"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112071045436948650">May 27, 2023 10:33 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Yeah it's different levels of granularity. I've generally found the less granular tests more useful</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:33 AM">(edited)</span></div></div></div></div><div id="chatlog__message-container-1112071157294829638" class="chatlog__message-container" data-message-id="1112071157294829638"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:33 AM">10:33</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">as a failure upstream will cause a large effect downstream in most cases</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:35 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112071350983606312" class="chatlog__message-container" data-message-id="1112071350983606312"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112071350983606312">May 27, 2023 10:34 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Echoing Andrew, the unit tests I wrote for Happenlance were: "Let me make sure i've implemented this memory-XOR function correctly in all cases". "Let me make sure I've written this packet compressor correctly without any dumb bugs". Those tests got #ifdef'd out and I think later deleted over the course of development! Their primary utility was aiding writing.</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:35 AM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thinking"><img class="chatlog__emoji chatlog__emoji--small" alt="🤔" src="files/1f914-15707.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112071588343459900" class="chatlog__message-container" data-message-id="1112071588343459900"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112071588343459900">May 27, 2023 10:35 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">another few factors in whether to write tests/what type to write are:
- is this an "artsy" or a "mathsy" problem I'm solving?
- am I writing to fit a pre-existing specification? (e.g. something that <em>has to</em> match with Linux's function call ABI)
- what does the dependency tree of this code look like?
  - small: this is close to a leaf of the call tree/1-off code
  - broad: lots of things directly depend on this (e.g. infrastructure code - allocators, error handling...)
  - deep: there's a long chain of dependencies on this thing (e.g. complex app-specific pipeline)s
- do I have an "Oracle" that can automatically tell me whether an answer is correct here?</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:41 AM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up_2"><img class="chatlog__emoji chatlog__emoji--small" alt="👆" src="files/1f446-9CC34.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112071703711977565" class="chatlog__message-container" data-message-id="1112071703711977565"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112071703711977565">May 27, 2023 10:35 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">root vs. leaf is definitely something I think about when testing</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112071625362382971" class="chatlog__message-container" data-message-id="1112071625362382971"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112069733806768209')"> I look at it as the following costs vs benefits
costs
- writing the test
- writing the code so that it is testable
- creating paraphernalia (mocks etc.)
- running the test (every build?)
benefits
- verifying alleged invariants so you can detect breaking changes at build instead of run time </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112071625362382971">May 27, 2023 10:35 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The benefits I find from testing:
* More reliable software. It's not clear that testing reduces bugs. But it certainly reduces regressions if you write a test whenever you find a bug.
* Better design. Any time spent making code easier to test results in moving towards the ideal of decoupled code.
* Documentation. It's hard to write documentation and keep it up to date. When I'm curious about something that I can't find docs for, tests (along with version control history) is a great source of breadcrumbs on the intent behind a module or subsystem. There are other reasons to write tests, so they often get done even when docs don't. And once they're done they're more likely to stay up to date than documentation.

The big cost of testing for me is not the time spent writing the test, that's negligible. Time spent making the code testable is a benefit, not a cost. The big cost is writing the wrong tests, that are dead weight or flaky.

To back up a bit, programming in my experience is not an amorphous activity where you're moving a knob from 88% to 90%. It feels like a black and white activity. Either you end up somewhere interesting or you don't. Time matters for this, but it's a satisficing rather than optimizing criterion. You get no points for getting somewhere uninteresting fast. Tests for me increase the odds of getting somewhere interesting. My path becomes more intentional where it used to be more like Brownian motion.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="open_mouth"><img class="chatlog__emoji chatlog__emoji--small" alt="😮" src="files/1f62e-FE396.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112072568220946545" class="chatlog__message-container" data-message-id="1112072568220946545"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112072568220946545">May 27, 2023 10:39 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">What about during more exploratory phases of development? Do tests still benefit you there? Seems like writing tests that may quickly become irrelevant as the program requirements change would just be wasted effort.</span></div></div></div></div><div id="chatlog__message-container-1112072637150138469" class="chatlog__message-container" data-message-id="1112072637150138469"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:39 AM">10:39</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I suppose that is what you mean by "writing the wrong tests"</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:39 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112072728493695017" class="chatlog__message-container" data-message-id="1112072728493695017"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112072728493695017">May 27, 2023 10:39 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The test I found the most valuable for our game was an input replay system - since our gameplay code already needed to be deterministic and isolated for networking to function, it was easy to save &amp; load input histories and resimulate the game at high speed to make sure nothing crashed, or to repro a glitch. Input replay is actually similar to how WhiteBox's integration testing works!</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112072846806630512" class="chatlog__message-container" data-message-id="1112072846806630512"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:40 AM">10:40</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">But it's worth pointing out that I wouldn't have been able to write that incredibly valuable integration test had our software architecture not already mandated a clean slice / hermetic seal of the gameplay tick code!</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:40 AM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="100"><img class="chatlog__emoji chatlog__emoji--small" alt="💯" src="files/1f4af-4CFF5.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112073035487379576" class="chatlog__message-container" data-message-id="1112073035487379576"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112073035487379576">May 27, 2023 10:41 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><div class="chatlog__markdown-quote"><div class="chatlog__markdown-quote-border"></div><div class="chatlog__markdown-quote-content">You get no points for getting somewhere uninteresting fast. Tests for me increase the odds of getting somewhere interesting. 
</div></div>purely as a personal experience question, this is the opposite of my experience</span></div></div></div></div><div id="chatlog__message-container-1112073089568751626" class="chatlog__message-container" data-message-id="1112073089568751626"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:41 AM">10:41</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I find testing to be, always, inertia</span></div></div></div></div><div id="chatlog__message-container-1112073122707931198" class="chatlog__message-container" data-message-id="1112073122707931198"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:41 AM">10:41</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">sometimes worthwhile inertia, but always inertia</span></div></div></div></div><div id="chatlog__message-container-1112073258641133718" class="chatlog__message-container" data-message-id="1112073258641133718"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:42 AM">10:42</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">none of the most creative projects I've ever worked on used testing seriously until very late in the lifecycle</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up_2"><img class="chatlog__emoji chatlog__emoji--small" alt="👆" src="files/1f446-9CC34.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="100"><img class="chatlog__emoji chatlog__emoji--small" alt="💯" src="files/1f4af-4CFF5.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112073327780057128" class="chatlog__message-container" data-message-id="1112073327780057128"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112073327780057128">May 27, 2023 10:42 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Agreed - I only wrote the tests I wrote because I didn't have to sledgehammer my own intuitive goals for the software's architecture just to avail the opportunity to test. I don't think I'll ever want to do that.</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:43 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112073552984817684" class="chatlog__message-container" data-message-id="1112073552984817684"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112073552984817684">May 27, 2023 10:43 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I find the value of testing increases significantly once you are actually shipping and pushing updates. The cost of bugs is much higher once it is affecting actual customers</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112073699013693554" class="chatlog__message-container" data-message-id="1112073699013693554"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112073699013693554">May 27, 2023 10:43 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><img loading="lazy" class="chatlog__emoji " alt="☝" title="☝" src="files/261d-23B11.svg">️ . It was only post-launch that I wrote a crash reporter for Happenlance, and not only was it incredibly useful, but it was fun, and I'm still proud of it :)</span></div></div></div></div><div id="chatlog__message-container-1112073729783111751" class="chatlog__message-container" data-message-id="1112073729783111751"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:43 AM">10:43</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I should've done it prelaunch!</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112073814843588628" class="chatlog__message-container" data-message-id="1112073814843588628"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112073258641133718')"> none of the most creative projects I've ever worked on used testing seriously until very late in the lifecycle </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112073814843588628">May 27, 2023 10:44 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">is "creative" similar to my use of "artsy" here (as juxtaposed against "mathsy")? Or do you mean more as "innovative"?</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112073892698271744" class="chatlog__message-container" data-message-id="1112073892698271744"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112073892698271744">May 27, 2023 10:44 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">yes, when you don't know the solution in advance</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112073990270373988" class="chatlog__message-container" data-message-id="1112073990270373988"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112073990270373988">May 27, 2023 10:45 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">"make me a GUI that makes it intuitive and fast to book flights"</span></div></div></div></div><div id="chatlog__message-container-1112074049602994266" class="chatlog__message-container" data-message-id="1112074049602994266"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:45 AM">10:45</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">99% of the value of that activity is trying many different ideas</span></div></div></div></div></div>
<div class=chatlog__message-group><div id=chatlog__message-container-1112100535391232061 class=chatlog__message-container data-message-id=1112100535391232061><div class=chatlog__message><div class=chatlog__message-aside><img class=chatlog__avatar src=files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png alt=Avatar loading=lazy></div><div class=chatlog__message-primary><div class=chatlog__header><span class=chatlog__author style=color:rgb(22,235,183) title="Kartik Agaram#9408" data-user-id=692826981640831107>Kartik Agaram</span> <span class=chatlog__timestamp><a href=#chatlog__message-container-1112100535391232061>May 27, 2023 12:30 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class=chatlog__markdown-preserve><span class="chatlog__markdown-mention" title="azmr#4870">@Andrew (azmr)</span> can you elaborate on how &quot;artsy vs mathsy&quot; might affect your approach to dev or testing?</span></div></div></div></div></div>
<div class=chatlog__message-group><div id=chatlog__message-container-1112103540094140446 class=chatlog__message-container data-message-id=1112103540094140446><div class=chatlog__message><div class=chatlog__message-aside><img class=chatlog__avatar src=files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png alt=Avatar loading=lazy></div><div class=chatlog__message-primary><div class=chatlog__header><span class=chatlog__author style=color:rgb(22,235,183) title=azmr#4870 data-user-id=240986595749134336>Andrew (azmr)</span> <span class=chatlog__timestamp><a href=#chatlog__message-container-1112103540094140446>May 27, 2023 12:42 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class=chatlog__markdown-preserve>there are a couple of components:
- requirements: precisely defined for &quot;mathsy&quot; - there is an objectively right and wrong answer; squishy for &quot;artsy&quot; - there may be a broad range of acceptable answers
- testability: mathsy stuff is often working with more primitive data, so is easier to generate &amp; evaluate tests for; artsy stuff may have to be visualized and subjectively evaluated in context
(&quot;artsy&quot; and &quot;mathsy&quot; are really 2 vague points in at least a spectrum, probably a multi-D space; they don&#39;t describe all possible categories)</span> <span class=chatlog__edited-timestamp title="05/27/2023 12:53 PM">(edited)</span></div><div class=chatlog__reactions><div class=chatlog__reaction title=heart><img class="chatlog__emoji chatlog__emoji--small" alt=❤️ src=files/2764-A3D25.svg loading=lazy> <span class=chatlog__reaction-count>2</span></div><div class=chatlog__reaction title=point_up_2><img class="chatlog__emoji chatlog__emoji--small" alt=👆 src=files/1f446-9CC34.svg loading=lazy> <span class=chatlog__reaction-count>1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112073955315028039" class="chatlog__message-container" data-message-id="1112073955315028039"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112073955315028039">May 27, 2023 10:44 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Crash reporting can be thought of as testing, but you're testing on the user's computer <img loading="lazy" class="chatlog__emoji " alt="🫣" title="🫣" src="files/1fae3-64122.svg"></span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="boom"><img class="chatlog__emoji chatlog__emoji--small" alt="💥" src="files/1f4a5-28689.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112074078329782363" class="chatlog__message-container" data-message-id="1112074078329782363"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112073955315028039')"> Crash reporting can be thought of as testing, but you're testing on the user's computer <img loading="lazy" class="chatlog__emoji " alt="🫣" title="🫣" src="files/1fae3-64122.svg"> </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112074078329782363">May 27, 2023 10:45 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">this is a very important part of the testing story at Mozilla actually</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="bangbang"><img class="chatlog__emoji chatlog__emoji--small" alt="‼️" src="files/203c-3876B.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112074097946529842" class="chatlog__message-container" data-message-id="1112074097946529842"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="Phil H#5516">Phil H</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112072568220946545')"> What about during more exploratory phases of development? Do tests still benefit you there? Seems like writing tests that may quickly become irrelevant as the program requirements change would just be wasted effort. </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112074097946529842">May 27, 2023 10:45 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Yeah there's definitely some art here. I'm not pro TDD. TDD is a good set of training wheels.

As a concrete example, when I published a text editor last year the 1.0 release had 0 tests. Eventually I started having enough bugs that I hate-wrote a bunch of tests. You can see the list here: <a href="https://lobste.rs/s/vsb6ue/case_for_models#c_d2mmvi" target="_blank">https://lobste.rs/s/vsb6ue/case_for_models#c_d2mmvi</a></span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112074381355651102" class="chatlog__message-container" data-message-id="1112074381355651102"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112074381355651102">May 27, 2023 10:46 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">In one sense, 1.0 is exactly the perfect time to write tests, because you know exactly what the product actually <em>is</em> and what's worthwhile, but it is also exactly the wrong time to write tests, because it's after a bunch of people hit a bunch of bugs <img loading="lazy" class="chatlog__emoji " alt="😅" title="sweat_smile" src="files/1f605-42B43.svg"></span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="100"><img class="chatlog__emoji chatlog__emoji--small" alt="💯" src="files/1f4af-4CFF5.svg" loading="lazy"> <span class="chatlog__reaction-count">3</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112074496279580785" class="chatlog__message-container" data-message-id="1112074496279580785"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112074496279580785">May 27, 2023 10:47 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">well "wrong" relative to what? in my framework it's just a question of how much time/effort you wasted</span></div></div></div></div><div id="chatlog__message-container-1112074544946094261" class="chatlog__message-container" data-message-id="1112074544946094261"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:47 AM">10:47</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">if you got to 1.0 faster, with more exploration, nothing was wrong or wasted</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112074686239625236" class="chatlog__message-container" data-message-id="1112074686239625236"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112074686239625236">May 27, 2023 10:47 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Right - but for something like a game, it's not ideal to have a bunch of customers who get a crash on startup <img loading="lazy" class="chatlog__emoji " alt="😳" title="flushed" src="files/1f633-5703D.svg"></span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112074753096826900" class="chatlog__message-container" data-message-id="1112074753096826900"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112074753096826900">May 27, 2023 10:48 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">winning a boxing match isn't a question of never taking a hit: it's not taking hits that you don't need to take</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="boxing_glove"><img class="chatlog__emoji chatlog__emoji--small" alt="🥊" src="files/1f94a-E5AB9.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112074804409933904" class="chatlog__message-container" data-message-id="1112074804409933904"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:48 AM">10:48</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">programming isn't about never having a bug</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="fax"><img class="chatlog__emoji chatlog__emoji--small" alt="📠" src="files/1f4e0-936A5.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112074810730754169" class="chatlog__message-container" data-message-id="1112074810730754169"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:48 AM">10:48</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">etc.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112075139790684311" class="chatlog__message-container" data-message-id="1112075139790684311"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112075139790684311">May 27, 2023 10:49 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">yeah bugs are inevitable. Minimizing the potential damage, and the effort spent finding and fixing bugs are my goals when testing</span> <span class="chatlog__edited-timestamp" title="05/27/2023 10:49 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112075244103008307" class="chatlog__message-container" data-message-id="1112075244103008307"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112075244103008307">May 27, 2023 10:49 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">yeah, we <em>certainly</em> didn't write any tests at all to my knowledge during the initial prototyping phases of development - that seems to me like a huge mistake. To explore is to travel light!</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112075501683605604" class="chatlog__message-container" data-message-id="1112075501683605604"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112075501683605604">May 27, 2023 10:51 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The only times when I feel like testing during active development is if I have to write something particularly tricky, e.g. a data structure</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="evergreen_tree"><img class="chatlog__emoji chatlog__emoji--small" alt="🌲" src="files/1f332-2DE37.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112075587712995499" class="chatlog__message-container" data-message-id="1112075587712995499"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:51 AM">10:51</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I've been burned many times by poor implementations that break all the interesting stuff I want to do</span></div></div></div></div><div id="chatlog__message-container-1112075749516652615" class="chatlog__message-container" data-message-id="1112075749516652615"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:52 AM">10:52</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I guess broadly a chunk of my testing is just "I want to run a small piece of this program right here and now"</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up_2"><img class="chatlog__emoji chatlog__emoji--small" alt="👆" src="files/1f446-9CC34.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112075765605994617" class="chatlog__message-container" data-message-id="1112075765605994617"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112075765605994617">May 27, 2023 10:52 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Also for refactoring imo. If I have a system that works as expected and I need to re-write it or swap out a dependency, having a suite of tests to verify that the new implementation works the same saves a ton of time</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up_2"><img class="chatlog__emoji chatlog__emoji--small" alt="👆" src="files/1f446-9CC34.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="heart"><img class="chatlog__emoji chatlog__emoji--small" alt="❤️" src="files/2764-A3D25.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class=chatlog__message-group><div id=chatlog__message-container-1112065247113658442 class=chatlog__message-container data-message-id=1112065247113658442><div class=chatlog__message><div class=chatlog__message-aside><img class=chatlog__avatar src=files/887ec965c6c9d5562ebdb09bf856ecea-B7764.png alt=Avatar loading=lazy></div><div class=chatlog__message-primary><div class=chatlog__header><span class=chatlog__author title=strager#1205 data-user-id=345398536550088705>strager (audience)</span> <span class=chatlog__timestamp><a href=#chatlog__message-container-1112065247113658442>May 27, 2023 10:10 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class=chatlog__markdown-preserve>My take: If I don&#39;t test my code, how do I know it works? (I don&#39;t.) If I don&#39;t know if it works, why did I bother writing it?

To me, testing isn&#39;t just automated testing. Manual testing is testing too. I either manually test, automatically test, or both. No testing is not an option.</span></div><div class=chatlog__reactions><div class=chatlog__reaction title=100><img class="chatlog__emoji chatlog__emoji--small" alt=💯 src=files/1f4af-4CFF5.svg loading=lazy> <span class=chatlog__reaction-count>2</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112075822434635827" class="chatlog__message-container" data-message-id="1112075822434635827"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112074686239625236')"> Right - but for something like a game, it's not ideal to have a bunch of customers who get a crash on startup <img loading="lazy" class="chatlog__emoji " alt="😳" title="flushed" src="files/1f633-5703D.svg"> </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112075822434635827">May 27, 2023 10:52 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Manual testing helps there, as <span class="chatlog__markdown-mention" title="strager#1205">@strager</span> mentioned in the audience. Even though I had 0 tests, I had a long list of manual tests I was running through on a daily basis. (I focused on figuring out an automated test framework when the list exploded 3x.)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112076085237121197" class="chatlog__message-container" data-message-id="1112076085237121197"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112076085237121197">May 27, 2023 10:53 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">good point - by far my most highly valued testing harness is of course Building And Running The Thing In The Debugger Every 2 Minutes</span></div></div></div></div><div id="chatlog__message-container-1112076214799183962" class="chatlog__message-container" data-message-id="1112076214799183962"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 10:53 AM">10:53</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">It's part of why having fast build times is super valuable - iteration, exploration, poking, prodding, stressing</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="100"><img class="chatlog__emoji chatlog__emoji--small" alt="💯" src="files/1f4af-4CFF5.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112076336899563560" class="chatlog__message-container" data-message-id="1112076336899563560"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112076214799183962')"> It's part of why having fast build times is super valuable - iteration, exploration, poking, prodding, stressing </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112076336899563560">May 27, 2023 10:54 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">yes and fast build is sometimes in conflict with testing</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112076415270142064" class="chatlog__message-container" data-message-id="1112076415270142064"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112076415270142064">May 27, 2023 10:54 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">True. Testing is slow!</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112076609323802664" class="chatlog__message-container" data-message-id="1112076609323802664"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112076609323802664">May 27, 2023 10:55 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">here's a thought experiment: how many bugs would your compiler have to magically catch automatically for you to accept it being 10x slower? 2x slower? etc.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112076705310445608" class="chatlog__message-container" data-message-id="1112076705310445608"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112076705310445608">May 27, 2023 10:55 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Just don't ask that question to Rust programmers <img loading="lazy" class="chatlog__emoji " alt="😏" title="smirk" src="files/1f60f-2BDEE.svg"></span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112076825817002004" class="chatlog__message-container" data-message-id="1112076825817002004"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112076085237121197')"> good point - by far my most highly valued testing harness is of course Building And Running The Thing In The Debugger Every 2 Minutes </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112076825817002004">May 27, 2023 10:56 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><em>ahem</em> or having it running live alongside your code <img loading="lazy" class="chatlog__emoji " alt="😇" title="innocent" src="files/1f607-0FB5B.svg"></span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="black_square_button"><img class="chatlog__emoji chatlog__emoji--small" alt="🔲" src="files/1f532-B0BE4.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112077100355174492" class="chatlog__message-container" data-message-id="1112077100355174492"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112077100355174492">May 27, 2023 10:57 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The speed of testing will be a good topic to explore I think, because there are some significant workflow and test-quality improvements that come with faster tests</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112077582880485439" class="chatlog__message-container" data-message-id="1112077582880485439"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112076609323802664')"> here's a thought experiment: how many bugs would your compiler have to magically catch automatically for you to accept it being 10x slower? 2x slower? etc. </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112077582880485439">May 27, 2023 10:59 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I think this is also an important question because you need to be actually catching bugs to make the time worthwhile</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112077589931114576" class="chatlog__message-container" data-message-id="1112077589931114576"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112076336899563560')"> yes and fast build is sometimes in conflict with testing </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112077589931114576">May 27, 2023 10:59 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">When I was a li'l baby programmer writing my own math.h library, I had a function that unit-tested each function on all 4 billion floats, and it caught a lot of problem inputs that were far away from zero, but it was annoying to wait for!
I got a more intuitive sense of my code, and more immediate value, out of writing a graphing program that let me view a graph of each function and sample the value of the function at the mouse cursor.
Both were useful, just on different timescales! I'd run the 4-billion-float test, trigger a bug, then seek to that coordinate in the graph to see what's goin on</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="heartpulse"><img class="chatlog__emoji chatlog__emoji--small" alt="💗" src="files/1f497-C1894.svg" loading="lazy"> <span class="chatlog__reaction-count">3</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112078433934123079" class="chatlog__message-container" data-message-id="1112078433934123079"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112077100355174492')"> The speed of testing will be a good topic to explore I think, because there are some significant workflow and test-quality improvements that come with faster tests </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112078433934123079">May 27, 2023 11:02 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">This is a good point. The tradeoff to Building and Running The Thing is that it's fast and easy, but you also tend to get stuck in a rut of testing things the exact same way every time, missing problems that automated testing would have caught -- as WhiteBox's integration testing system has proven to me time and again <img loading="lazy" class="chatlog__emoji " alt="😅" title="sweat_smile" src="files/1f605-42B43.svg"></span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112078514011766824" class="chatlog__message-container" data-message-id="1112078514011766824"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112078514011766824">May 27, 2023 11:02 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">As has already been mentioned, adding tests adds friction to changing the code under test, particularly if changing APIs. For WhiteBox code this seems to manifest in us unit testing the leaves of the call tree/infrastructure code (which operates on simple data types), and the root of the tree (fake user input), but basically nothing in the middle levels.  Those are left to either asserts or the full-program tests to surface errors. I find that the mid-levels are where most of the code/API volatility is, as new requirements call for refactoring/restructuring.
This sandwich structure basically means that writing tests is O(1) rather than O(n_code_edits)</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:05 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112078900017758388" class="chatlog__message-container" data-message-id="1112078900017758388"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112078900017758388">May 27, 2023 11:04 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">That's funny; in my game dev experience it seems like a lot of my testing was smack in the middle of the sandwich!</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:05 AM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thinking"><img class="chatlog__emoji chatlog__emoji--small" alt="🤔" src="files/1f914-15707.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="leafy_green"><img class="chatlog__emoji chatlog__emoji--small" alt="🥬" src="files/1f96c-A9ABB.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112079267522695218" class="chatlog__message-container" data-message-id="1112079267522695218"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112079267522695218">May 27, 2023 11:05 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Let's talk more about the costs and benefits of testing - I think that will be helpful because a lot of people underestimate the costs and overestimate the benefits</span></div></div></div></div><div id="chatlog__message-container-1112079351752708136" class="chatlog__message-container" data-message-id="1112079351752708136"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:06 AM">11:06</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">For costs, Demetri already mentioned inertia, but I think there are several facets to that</span></div></div></div></div><div id="chatlog__message-container-1112079477737013338" class="chatlog__message-container" data-message-id="1112079477737013338"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:06 AM">11:06</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">There's dev time, obviously, although it seems some developers (perhaps Kartik for example?) have workflows that lend themselves to lots of testing on the fly</span></div></div></div></div><div id="chatlog__message-container-1112079525199757373" class="chatlog__message-container" data-message-id="1112079525199757373"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:07 AM">11:07</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">compile time and run time too, obviously</span></div></div></div></div><div id="chatlog__message-container-1112079553104465931" class="chatlog__message-container" data-message-id="1112079553104465931"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:07 AM">11:07</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">but are there others?</span></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112079742124957747" class="chatlog__message-container" data-message-id="1112079742124957747"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112079553104465931')"> but are there others? </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112079742124957747">May 27, 2023 11:07 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">procrustean costs (having to make your code testable) and paraphernalia costs (mocks or whatever)</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:08 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112079758063304704" class="chatlog__message-container" data-message-id="1112079758063304704"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112079758063304704">May 27, 2023 11:07 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Unit testing seems like the highest inertia - you are writing multiple nontrivial functions, that (hopefully!) cover a huge input range, for each little quantum of your software</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112079853672468481" class="chatlog__message-container" data-message-id="1112079853672468481"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112079742124957747')"> procrustean costs (having to make your code testable) and paraphernalia costs (mocks or whatever) </span> <span class="chatlog__reply-edited-timestamp" title="05/27/2023 11:08 AM">(edited)</span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112079853672468481">May 27, 2023 11:08 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">you should go more into what it takes to "make your code testable"</span></div></div></div></div><div id="chatlog__message-container-1112079884429295787" class="chatlog__message-container" data-message-id="1112079884429295787"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:08 AM">11:08</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">many people regard it as an unconditional good to do so</span></div></div></div></div><div id="chatlog__message-container-1112079925969686661" class="chatlog__message-container" data-message-id="1112079925969686661"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:08 AM">11:08</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">you seemingly do not <img loading="lazy" class="chatlog__emoji " alt="🙂" title="slight_smile" src="files/1f642-83E8A.svg"></span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112079932143702136" class="chatlog__message-container" data-message-id="1112079932143702136"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112079932143702136">May 27, 2023 11:08 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><em>being testable</em> is unconditionally good</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div><div id="chatlog__message-container-1112079951814996109" class="chatlog__message-container" data-message-id="1112079951814996109"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:08 AM">11:08</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">but you don't get it for free</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112080125366911066" class="chatlog__message-container" data-message-id="1112080125366911066"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:09 AM">11:09</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">testability requires "points of legibility" where you can extract state snapshots to be compared against reference cases</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div><div id="chatlog__message-container-1112080156346023999" class="chatlog__message-container" data-message-id="1112080156346023999"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:09 AM">11:09</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">but general coding does not require that</span></div></div></div></div></div><div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112079589200629932" class="chatlog__message-container" data-message-id="1112079589200629932"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112078900017758388')"> That's funny; in my game dev experience it seems like a lot of my testing was smack in the middle of the sandwich! </span> <span class="chatlog__reply-edited-timestamp" title="05/27/2023 11:05 AM">(edited)</span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112079589200629932">May 27, 2023 11:07 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">oh interesting! Did you have the same volatility layout there?</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112080369190187179" class="chatlog__message-container" data-message-id="1112080369190187179"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112080369190187179">May 27, 2023 11:10 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">the code <em>inside</em> the gameplay tick was volatile, and the code <em>outside</em> the gameplay tick was volatile, but having the gasket between those two layers was a constant - so maybe it's the slice of lettuce near the top of the sandwich instead of the bread per se <img loading="lazy" class="chatlog__emoji " alt="🤪" title="zany_face" src="files/1f92a-55954.svg"></span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112080507115683872" class="chatlog__message-container" data-message-id="1112080507115683872"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112080507115683872">May 27, 2023 11:10 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">so yeah, in Demetri's terms, we had that point of legibility</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112080460244332674" class="chatlog__message-container" data-message-id="1112080460244332674"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112080460244332674">May 27, 2023 11:10 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I've said some of this already, but to lay it out I see the following costs of testing:
* False-positive tests that fail flakily when nothing is actually wrong.
* Brittle tests where the API keeps changing, as <span class="chatlog__markdown-mention" title="azmr#4870">@Andrew (azmr)</span> mentioned. In large teams it's easy to lose one's bearings and write tests that aren't useful, or keep tests around past their usefulness. Those tend to be the hardest decisions, what tests to write and when to throw away a test.
* This isn't a drawback of tests, exactly, but the biggest way tests fail to help is by being difficult to write in many situations. Graphics, non-determinism and simulation are some domains where I've been forced to muddle along without tests, and where others who force themselves to write tests often end up with less-than-useful tests.</span></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112080829527638118" class="chatlog__message-container" data-message-id="1112080829527638118"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112080125366911066')"> testability requires "points of legibility" where you can extract state snapshots to be compared against reference cases </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112080829527638118">May 27, 2023 11:12 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">where do you see this causing problems?</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112080864579436687" class="chatlog__message-container" data-message-id="1112080864579436687"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112080507115683872')"> so yeah, in Demetri's terms, we had that point of legibility </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112080864579436687">May 27, 2023 11:12 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I think having tests concentrated at "narrow interfaces between larger components" is good, and it lines up with my preference for log-file debugging</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112080870124310653" class="chatlog__message-container" data-message-id="1112080870124310653"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112080870124310653">May 27, 2023 11:12 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Re: Overestimating the benefits of tests
Some things are inherently hard to test. Most of the nasty bugs I dealt with for live services are ones that did not manifest until deployed to a production environment. For example, during a deployment servers would have a mix of build V1 and the new build V2. A user would make a request to V2 and write new binary data to storage. The subsequent request might happen to try to retrieve that data but using a server with build V1 as build V2 was still rolling out. V1 has no knowledge of the new format and would fail to retrieve the data. Without writing a complex testing system to verify a user using a mix of builds, this is a case where a developer just needs to be aware of the consequences of using a new memory layout.

Other bugs also followed this pattern of unanticipated scenarios such as operation under high load, multithreading bugs, and unexpected user data.</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:12 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112081001473114152" class="chatlog__message-container" data-message-id="1112081001473114152"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112081001473114152">May 27, 2023 11:12 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Ohhhhh yes, I have found graphics to be hard to test. That could easily send me on a tangent about using <em>external tooling</em> for testing, which can be a dream or a nightmare of its own.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112081040161390732" class="chatlog__message-container" data-message-id="1112081040161390732"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112079742124957747')"> procrustean costs (having to make your code testable) and paraphernalia costs (mocks or whatever) </span> <span class="chatlog__reply-edited-timestamp" title="05/27/2023 11:08 AM">(edited)</span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112081040161390732">May 27, 2023 11:13 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I disagree with this comment but <em>love</em> the metaphor <img loading="lazy" class="chatlog__emoji " alt="🙂" title="slight_smile" src="files/1f642-83E8A.svg"> Code having to stretch or chop because of some external constraint is :chefkiss:</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112081152128319549" class="chatlog__message-container" data-message-id="1112081152128319549"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112080829527638118')"> where do you see this causing problems? </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112081152128319549">May 27, 2023 11:13 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">well if the invariant you want to test is not manifested anywhere in the code, i.e. is an emergent property, now you have to manifest it somehow</span></div></div></div></div><div id="chatlog__message-container-1112081405967605820" class="chatlog__message-container" data-message-id="1112081405967605820"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:14 AM">11:14</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">if every test were of the form "verify x &gt;= 0" then we wouldn't have this conversation</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112081538713124984" class="chatlog__message-container" data-message-id="1112081538713124984"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408">Kartik Agaram</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112080460244332674')"> I've said some of this already, but to lay it out I see the following costs of testing:
* False-positive tests that fail flakily when nothing is actually wrong.
* Brittle tests where the API keeps changing, as <span class="chatlog__markdown-mention" title="azmr#4870">@Andrew (azmr)</span> mentioned. In large teams it's easy to lose one's bearings and write tests that aren't useful, or keep tests around past their usefulness. Those tend to be the hardest decisions, what tests to write and when to throw away a test.
* This isn't a drawback of tests, exactly, but the biggest way tests fail to help is by being difficult to write in many situations. Graphics, non-determinism and simulation are some domains where I've been forced to muddle along without tests, and where others who force themselves to write tests often end up with less-than-useful tests. </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112081538713124984">May 27, 2023 11:15 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><div class="chatlog__markdown-quote"><div class="chatlog__markdown-quote-border"></div><div class="chatlog__markdown-quote-content">False-positive tests that fail flakily when nothing is actually wrong.
</div></div>This reminds me that I haven't discussed asserts!! <img loading="lazy" class="chatlog__emoji " alt="😁" title="grin" src="files/1f601-4E9BD.svg"></span></div></div></div></div><div id="chatlog__message-container-1112081573295165452" class="chatlog__message-container" data-message-id="1112081573295165452"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:15 AM">11:15</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">speaking of verify(x &gt;= 0), haha</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112081628899061761" class="chatlog__message-container" data-message-id="1112081628899061761"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112081628899061761">May 27, 2023 11:15 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">we definitely have some overzealous asserts <img loading="lazy" class="chatlog__emoji " alt="😅" title="sweat_smile" src="files/1f605-42B43.svg"></span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112081821686046851" class="chatlog__message-container" data-message-id="1112081821686046851"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112081821686046851">May 27, 2023 11:16 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Yeah, don't let my lack of interest in unit tests imply that I'm not a huge assert zealot</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112082001483280494" class="chatlog__message-container" data-message-id="1112082001483280494"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112082001483280494">May 27, 2023 11:16 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I don't really think of most of my runtime/compile-time asserts as "tests". Rather just verifications of the assumptions I am making in code</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:17 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112082068717977640" class="chatlog__message-container" data-message-id="1112082068717977640"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112079853672468481')"> you should go more into what it takes to "make your code testable" </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112082068717977640">May 27, 2023 11:17 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">to elaborate on this, what test do you propose for verifying that your hash function mixes well on your data set? you can do it, of course, but it's not something you can just say x&gt;=0 from code you already have</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112082405902270595" class="chatlog__message-container" data-message-id="1112082405902270595"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112082405902270595">May 27, 2023 11:18 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Debugging and testing have been brought up as distinct, but a large benefit I've found from our integration tests is to repeatedly get the program to a state where something interesting is happening that needs inspection in a debugger without having to manipulate the GUI by hand each time.
This really helps to keep all of the problem details and hypotheses in my working memory at the right LOD without evicting them from cache with "and then I have to click this over here...".</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:20 AM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up_2"><img class="chatlog__emoji chatlog__emoji--small" alt="👆" src="files/1f446-9CC34.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div><div class="chatlog__reaction" title="heart"><img class="chatlog__emoji chatlog__emoji--small" alt="❤️" src="files/2764-A3D25.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112082412650893353" class="chatlog__message-container" data-message-id="1112082412650893353"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112080125366911066')"> testability requires "points of legibility" where you can extract state snapshots to be compared against reference cases </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112082412650893353">May 27, 2023 11:18 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I certainly don't think it's worth making every single line of code testable. Rather the ideal is to have a large project criss-crossed with places where we're cross-checking answers from two sources (tests are really just a second source, along with types and probably other approaches).

When I write large projects it's easy to forget to keep this mesh network of cross-checks developing alongside features. Tests remind me to maintain this mesh network.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112082421807054988" class="chatlog__message-container" data-message-id="1112082421807054988"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112082421807054988">May 27, 2023 11:18 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">as for whether procrustean costs exist, I'm not sure I understand the potential disagreement ... surely we all agree that there is code that is not testable, and it can be (maybe) made testable?</span></div></div></div></div><div id="chatlog__message-container-1112082488932700291" class="chatlog__message-container" data-message-id="1112082488932700291"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:18 AM">11:18</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">(and that takes effort and design, which is a procrustean cost)</span></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112082919968739448" class="chatlog__message-container" data-message-id="1112082919968739448"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112082919968739448">May 27, 2023 11:20 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The input space might be so large that it's infeasible to test every possible case</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112082966202548254" class="chatlog__message-container" data-message-id="1112082966202548254"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112082966202548254">May 27, 2023 11:20 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">yeah, with agile-ish / UX-heavy / "goopy" programming, it's very easy to make a lot of code that is very not-systematically-testable, but you're doing that because you want to get to interesting places</span></div></div></div></div><div id="chatlog__message-container-1112083109261877338" class="chatlog__message-container" data-message-id="1112083109261877338"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:21 AM">11:21</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">at that point, you either bust out the sledgehammer, or you fall back to wide wide system testing (crash reporter, QA teams, etc)</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:22 AM">(edited)</span></div></div></div></div><div id="chatlog__message-container-1112083186881662996" class="chatlog__message-container" data-message-id="1112083186881662996"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:21 AM">11:21</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">you just shoot the rocket at the moon 10 times and see if you get there on the 11th</span></div></div></div></div><div id="chatlog__message-container-1112083577115529369" class="chatlog__message-container" data-message-id="1112083577115529369"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:23 AM">11:23</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">There's no formal proof that the Saturn V can make it to orbit, etc</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112083829633597500" class="chatlog__message-container" data-message-id="1112083829633597500"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112083829633597500">May 27, 2023 11:24 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Re procrustean costs: one way that the WhiteBox codebase lucked out with integration testing is that we had to make our user actions/inputs data-driven so that it could be controlled by an external editor. This meant that our test "harness"(?) basically just acts as another editor sending it commands (really just pushing them into a queue).
If we didn't already have this constraint and instead had input handling done inline with other GUI code, I'm not sure how I would have felt about refactoring to force this structure just for testing...</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112084089592356935" class="chatlog__message-container" data-message-id="1112084089592356935"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112084089592356935">May 27, 2023 11:25 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Hey, that's another example of tests organically emerging out of an architectural boundary that was already being imposed!</span></div></div></div></div><div id="chatlog__message-container-1112084198027694120" class="chatlog__message-container" data-message-id="1112084198027694120"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:25 AM">11:25</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Tests are like oysters.</span></div></div></div></div><div id="chatlog__message-container-1112084337731588146" class="chatlog__message-container" data-message-id="1112084337731588146"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:26 AM">11:26</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">They grow where the water meets the shore</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112084349093945507" class="chatlog__message-container" data-message-id="1112084349093945507"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112082068717977640')"> to elaborate on this, what test do you propose for verifying that your hash function mixes well on your data set? you can do it, of course, but it's not something you can just say x&gt;=0 from code you already have </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112084349093945507">May 27, 2023 11:26 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">what do you do in these situations to verify that your hash function is doing its job well?</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:26 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112084502395760702" class="chatlog__message-container" data-message-id="1112084502395760702"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112084502395760702">May 27, 2023 11:26 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I have to build an entire separate testing system to generate inputs and log outputs and perform statistical checks, i.e. code that was not at all necessary for the functioning of the application</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112084544066175007" class="chatlog__message-container" data-message-id="1112084544066175007"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112084544066175007">May 27, 2023 11:26 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">gotcha</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112084658939760721" class="chatlog__message-container" data-message-id="1112084658939760721"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112084658939760721">May 27, 2023 11:27 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">this is also why "just use murmur3" became a refrain for me :)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112084776145408140" class="chatlog__message-container" data-message-id="1112084776145408140"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112084502395760702')"> I have to build an entire separate testing system to generate inputs and log outputs and perform statistical checks, i.e. code that was not at all necessary for the functioning of the application </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112084776145408140">May 27, 2023 11:27 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">This is one reason I love the domain of games: there's a huge culture of building out huge frameworks of tools. Other industry verticals could learn a thing or two.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112084817916477462" class="chatlog__message-container" data-message-id="1112084817916477462"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112084817916477462">May 27, 2023 11:28 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">That's reminiscent of what happens in profiling, btw - which I suppose is a narrow special-case form of testing</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112084927945646211" class="chatlog__message-container" data-message-id="1112084927945646211"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112084927945646211">May 27, 2023 11:28 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">well, it's a form of measurement at least, although I did did once write a test called <code class="chatlog__markdown-pre chatlog__markdown-pre--inline">it('should take less than 1000 years to run this code')</code></span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="joy"><img class="chatlog__emoji chatlog__emoji--small" alt="😂" src="files/1f602-168C5.svg" loading="lazy"> <span class="chatlog__reaction-count">3</span></div><div class="chatlog__reaction" title="heart"><img class="chatlog__emoji chatlog__emoji--small" alt="❤️" src="files/2764-A3D25.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112085200193728623" class="chatlog__message-container" data-message-id="1112085200193728623"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112084927945646211')"> well, it's a form of measurement at least, although I did did once write a test called <code class="chatlog__markdown-pre chatlog__markdown-pre--inline">it('should take less than 1000 years to run this code')</code> </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112085200193728623">May 27, 2023 11:29 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Some of my tests don't have any assertions, just a comment <code class="chatlog__markdown-pre chatlog__markdown-pre--inline"># shouldn't crash</code>.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="boom"><img class="chatlog__emoji chatlog__emoji--small" alt="💥" src="files/1f4a5-28689.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112085345782218864" class="chatlog__message-container" data-message-id="1112085345782218864"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112085345782218864">May 27, 2023 11:30 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Valid form of testing!</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112085449406685256" class="chatlog__message-container" data-message-id="1112085449406685256"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112085449406685256">May 27, 2023 11:30 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">right, you always hear about unit tests and integration tests, but why not unit <em>benchmarks</em> and integration <em>benchmarks</em>?! I know some companies will autofail commits with a certain % slowdown for given inputs</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="open_mouth"><img class="chatlog__emoji chatlog__emoji--small" alt="😮" src="files/1f62e-FE396.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">3</span></div><div class="chatlog__reaction" title="gottagofast"><img class="chatlog__emoji chatlog__emoji--small" alt="gottagofast" src="files/882479864475697173-D0C04.png" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="chart_with_downwards_trend"><img class="chatlog__emoji chatlog__emoji--small" alt="📉" src="files/1f4c9-E3D65.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div><div id="chatlog__message-container-1112085497490190406" class="chatlog__message-container" data-message-id="1112085497490190406"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:30 AM">11:30</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">speed is a form of behaviour...</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112085568516538419" class="chatlog__message-container" data-message-id="1112085568516538419"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112085568516538419">May 27, 2023 11:31 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">We had that kind of live monitoring for our services. On call engineers would be woken up if latency was too high</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="chart_with_upwards_trend"><img class="chatlog__emoji chatlog__emoji--small" alt="📈" src="files/1f4c8-E041F.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div><div class="chatlog__reaction" title="fire"><img class="chatlog__emoji chatlog__emoji--small" alt="🔥" src="files/1f525-8FE4F.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112085723483480095" class="chatlog__message-container" data-message-id="1112085723483480095"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:31 AM">11:31</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">And we wouldn't deploy new builds if the test servers exhibited different latency behavior</span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112086133128568964" class="chatlog__message-container" data-message-id="1112086133128568964"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086133128568964">May 27, 2023 11:33 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Yeah, once you get into distributed system testing (multi-server/multi-thread/concurrency) then testing becomes a LOT harder!</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112085860293283891" class="chatlog__message-container" data-message-id="1112085860293283891"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112085860293283891">May 27, 2023 11:32 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">So, when I think back on the testing at my web dev job, I think the tests had really high costs for next to zero benefit. We had some kind of code coverage requirement, which we achieved, but the tests were almost all using this "snapshot" feature of the Jest JS testing framework, where the framework just saves the output of your program to a file and later asserts that the output is exactly the same.</span></div></div></div></div><div id="chatlog__message-container-1112086035577458688" class="chatlog__message-container" data-message-id="1112086035577458688"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:32 AM">11:32</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">This led to the insane false positive rate I mentioned before, because the data being snapshotted was, like, the full DOM produced by a React component</span></div></div></div></div><div id="chatlog__message-container-1112086080209043486" class="chatlog__message-container" data-message-id="1112086080209043486"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:33 AM">11:33</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">adding a single CSS class would start failing tests</span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112086302549090426" class="chatlog__message-container" data-message-id="1112086302549090426"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086302549090426">May 27, 2023 11:33 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">to combat this, Jest had a convenient feature which would automatically update all your snapshot files to stop failing</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="grimacing"><img class="chatlog__emoji chatlog__emoji--small" alt="😬" src="files/1f62c-CE43C.svg" loading="lazy"> <span class="chatlog__reaction-count">6</span></div><div class="chatlog__reaction" title="white_check_mark"><img class="chatlog__emoji chatlog__emoji--small" alt="✅" src="files/2705-0589F.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div><div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112086392365908028" class="chatlog__message-container" data-message-id="1112086392365908028"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086392365908028">May 27, 2023 11:34 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">these tests took almost an hour to run and clearly have actually zero benefit</span></div></div></div></div></div><div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112086551766237264" class="chatlog__message-container" data-message-id="1112086551766237264"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086551766237264">May 27, 2023 11:34 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">but to me it clearly demonstrates a problem with mere "coverage"</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div><div id="chatlog__message-container-1112086646536536186" class="chatlog__message-container" data-message-id="1112086646536536186"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:35 AM">11:35</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I guess the best you could say is that we had it-doesn't-crash coverage of 70% or so of our code?</span></div></div></div></div><div id="chatlog__message-container-1112086663200510105" class="chatlog__message-container" data-message-id="1112086663200510105"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:35 AM">11:35</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">but that's being generous</span></div></div></div></div></div><div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112087001391431742" class="chatlog__message-container" data-message-id="1112087001391431742"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112086551766237264')"> but to me it clearly demonstrates a problem with mere "coverage" </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112087001391431742">May 27, 2023 11:36 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Ugh, "coverage"</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112086124727369748" class="chatlog__message-container" data-message-id="1112086124727369748"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="azmr#4870">Andrew (azmr)</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112085497490190406')"> speed is a form of behaviour... </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086124727369748">May 27, 2023 11:33 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Absolutely. But autofailing on x% slowdown is a blunt instrument. Chrome did this since their launch. The result was that the app grew slower with a slope of x%.

In the past one approach I've found for this is to write performance tests in a white-box (ha!) manner. I count some sort of metric while running a test, and verify afterwards that it stays within some bound. For example a sort function may count swaps. Sometimes counting allocations can be useful. Still blunt, but less blunt.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="laughing"><img class="chatlog__emoji chatlog__emoji--small" alt="😆" src="files/1f606-BE94E.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>

<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112086346513776701" class="chatlog__message-container" data-message-id="1112086346513776701"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086346513776701">May 27, 2023 11:34 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I was talking to <span class="chatlog__markdown-mention" title="4984#4984">@Shaw</span> about FastVM a while back and I suggested autofailing commits on less-than-x% <em>speedup</em>. It's not a crazy idea!</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:34 AM">(edited)</span></div></div></div></div></div>


<div class="chatlog__message-group"><div id="chatlog__message-container-1112086418739703900" class="chatlog__message-container" data-message-id="1112086418739703900"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112086133128568964')"> Yeah, once you get into distributed system testing (multi-server/multi-thread/concurrency) then testing becomes a LOT harder! </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086418739703900">May 27, 2023 11:34 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">yes, again, why I have settled on log files and scripted log analysis as my main tool</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112087118966161528" class="chatlog__message-container" data-message-id="1112087118966161528"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src=files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class=chatlog__header><span class=chatlog__author style=color:rgb(22,235,183) title=demetrispanos#5588 data-user-id=239823533989953546>demetrispanos</span> <span class=chatlog__timestamp><a href=#chatlog__message-container-1112087118966161528>May 27, 2023 11:37 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class=chatlog__markdown-preserve>elaborating on this, you can write a script that says &quot;there&#39;s never an event of type B that happens outside a window of 50ms of an event of type A&quot;</span></div></div></div></div><div id=chatlog__message-container-1112087166126932029 class=chatlog__message-container data-message-id=1112087166126932029><div class=chatlog__message><div class=chatlog__message-aside><div class=chatlog__short-timestamp title="05/27/2023 11:37 AM">11:37</div></div><div class=chatlog__message-primary><div class="chatlog__content chatlog__markdown"><span class=chatlog__markdown-preserve>very annoying to do in conventional testing, trivial to do with logs</span></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112086664299417680" class="chatlog__message-container" data-message-id="1112086664299417680"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112086418739703900')"> yes, again, why I have settled on log files and scripted log analysis as my main tool </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086664299417680">May 27, 2023 11:35 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I originally described my white-box tests as making assertions on the log a test emitted. <a href="http://akkartik.name/post/tracing-tests" target="_blank">http://akkartik.name/post/tracing-tests</a></span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112086910098223145" class="chatlog__message-container" data-message-id="1112086910098223145"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086910098223145">May 27, 2023 11:36 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">yes, lines up very well with how I think about logging/testing</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112086766908882974" class="chatlog__message-container" data-message-id="1112086766908882974"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112086133128568964')"> Yeah, once you get into distributed system testing (multi-server/multi-thread/concurrency) then testing becomes a LOT harder! </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112086766908882974">May 27, 2023 11:35 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">That's why I found the most valuable testing to be fake user accounts making requests from other servers to verify that the actual deployed code worked with it's real dependencies. Never found much use in mocks. We also had a lot of process around rolling out new features slowly to  small subsets of users and always having monitoring and logging in place</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:36 AM">(edited)</span></div></div></div></div></div>


<div class="chatlog__message-group"><div id="chatlog__message-container-1112087018726490283" class="chatlog__message-container" data-message-id="1112087018726490283"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112087018726490283">May 27, 2023 11:36 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">and a really solid way to turn new features on and off if things went south</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112087030772539402" class="chatlog__message-container" data-message-id="1112087030772539402"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112086302549090426')"> to combat this, Jest had a convenient feature which would automatically update all your snapshot files to stop failing </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112087030772539402">May 27, 2023 11:36 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">This makes me think that the central failing of "test-orientation" is the impulse to write tests-as-rote. If you're not building testing systems intelligently and deliberately, I don't see any value in them!</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thinking"><img class="chatlog__emoji chatlog__emoji--small" alt="🤔" src="files/1f914-15707.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112087157172097074" class="chatlog__message-container" data-message-id="1112087157172097074"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112087157172097074">May 27, 2023 11:37 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">at the same time, <span class="chatlog__markdown-mention" title="Phil H#5516">@Phil H</span> was talking in coffee on Thursday about devs who would ship stuff without ever running it <em>at all</em></span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112087249581002884" class="chatlog__message-container" data-message-id="1112087249581002884"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112087249581002884">May 27, 2023 11:37 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Couldn't be me... <img loading="lazy" class="chatlog__emoji " alt="🫣" title="🫣" src="files/1fae3-64122.svg"></span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="eyes"><img class="chatlog__emoji chatlog__emoji--small" alt="👀" src="files/1f440-6C64D.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112087315016339468" class="chatlog__message-container" data-message-id="1112087315016339468"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112087315016339468">May 27, 2023 11:37 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">everybody knows that mere lines-of-code coverage is a flawed metric, but more broadly, I'm curious how people here think about how well their tests "cover" the system</span></div></div></div></div><div id="chatlog__message-container-1112087398197755924" class="chatlog__message-container" data-message-id="1112087398197755924"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:38 AM">11:38</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">for any system I test, I have some intuition for how much of the space is covered, even if 100% of the lines are covered</span></div></div></div></div><div id="chatlog__message-container-1112087498055757914" class="chatlog__message-container" data-message-id="1112087498055757914"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:38 AM">11:38</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">but is that feeling accurate and can it scale?</span></div></div></div></div><div id="chatlog__message-container-1112087557124149399" class="chatlog__message-container" data-message-id="1112087557124149399"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:38 AM">11:38</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">it seems to devolve into the worst kind of "what gets measured gets managed" in practice</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112087788041547908" class="chatlog__message-container" data-message-id="1112087788041547908"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112087315016339468')"> everybody knows that mere lines-of-code coverage is a flawed metric, but more broadly, I'm curious how people here think about how well their tests "cover" the system </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112087788041547908">May 27, 2023 11:39 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I've got nothin' here. I just go with my gut: <a href="http://akkartik.name/post/2009-01-24-19-46-26_002-soc" target="_blank">http://akkartik.name/post/2009-01-24-19-46-26_002-soc</a>. I started writing tests for my text editor last year when I started swearing at myself for my own incompetence.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112088151293444208" class="chatlog__message-container" data-message-id="1112088151293444208"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112088151293444208">May 27, 2023 11:41 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><span class="chatlog__markdown-mention" title="Phil H#5516">@Phil H</span> , you mentioned on Thursday that you would have weekly reviews of test and system failures - that might be relevant here?</span></div></div></div></div><div id="chatlog__message-container-1112088246046969886" class="chatlog__message-container" data-message-id="1112088246046969886"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:41 AM">11:41</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">what were those conversations like and what were the outcomes?</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112088660888801330" class="chatlog__message-container" data-message-id="1112088660888801330"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112088660888801330">May 27, 2023 11:43 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I would again emphasize the significance of narrow interfaces here, because testing there yields much more benefit than elsewhere</span></div></div></div></div><div id="chatlog__message-container-1112088752211365929" class="chatlog__message-container" data-message-id="1112088752211365929"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:43 AM">11:43</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">the purpose of "coverage" is to be able to localize problems to a "covered thing"</span></div></div></div></div><div id="chatlog__message-container-1112088889612578886" class="chatlog__message-container" data-message-id="1112088889612578886"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:44 AM">11:44</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">if you just have one test for your whole program, it tells you "there's a problem somewhere in the program, glhf"</span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112089030654439444" class="chatlog__message-container" data-message-id="1112089030654439444"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112089030654439444">May 27, 2023 11:44 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">so now you want your "covered things" to be useful attentional units</span></div></div></div></div><div id="chatlog__message-container-1112089065626554478" class="chatlog__message-container" data-message-id="1112089065626554478"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:44 AM">11:44</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">"there's a problem in the DB driver" ok cool I know where to start poking</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112088965835673802" class="chatlog__message-container" data-message-id="1112088965835673802"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112088965835673802">May 27, 2023 11:44 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><div class="chatlog__markdown-quote"><div class="chatlog__markdown-quote-border"></div><div class="chatlog__markdown-quote-content">the purpose of "coverage"
</div></div>now we're getting somewhere</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112088993388052540" class="chatlog__message-container" data-message-id="1112088993388052540"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112087398197755924')"> for any system I test, I have some intuition for how much of the space is covered, even if 100% of the lines are covered </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112088993388052540">May 27, 2023 11:44 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">it seems like a more apt (but harder to pin-down) metric is "coverage of potential state-space", which can be approximated as "coverage of potential inputs". This, I think, captures why I lean on root-level end-to-end integration tests: "are we looking right for the common and known-awkward user inputs?".
These will occasionally miss some important edge-cases (e.g. ring structures wrapping with a particular offset) that unit tests can then precisely target.</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:46 AM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112089067534950531" class="chatlog__message-container" data-message-id="1112089067534950531"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112089067534950531">May 27, 2023 11:44 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Yeah, and testing at the bottleneck - like the health inspector at the conveyor belt - lets you vet the entire subsystem</span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112089285882032258" class="chatlog__message-container" data-message-id="1112089285882032258"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112089285882032258">May 27, 2023 11:45 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">right exactly, if one test can eliminate 90% of the code as a possible source of problems, that is good</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112089194446192700" class="chatlog__message-container" data-message-id="1112089194446192700"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112088151293444208')"> <span class="chatlog__markdown-mention" title="Phil H#5516">@Phil H</span> , you mentioned on Thursday that you would have weekly reviews of test and system failures - that might be relevant here? </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112089194446192700">May 27, 2023 11:45 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Yeah we had weekly review of the health of the services. The focus would be around post-mortems for any severe incidents that impacted users. We would also look at peak server load to see if we needed to expand and other "key performance metrics" such as latency to see if there were any regressions. For any post-mortems there was usually a chain of things that went wrong for it to get to that point and we assign urgent tasks to cover the holes in our process and testing. It could be anywhere from writing tests, adding live monitoring for something that went un-noticed, adding a new manual verification step to our build verifications etc</span></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112089527109046272" class="chatlog__message-container" data-message-id="1112089527109046272"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112089527109046272">May 27, 2023 11:46 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">one of my robotics team parents was just telling me about changes to quality inspection in the early days of the automotive industry and now I wish I remembered who the people involved were</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:46 AM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112089948930183338" class="chatlog__message-container" data-message-id="1112089948930183338"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112088752211365929')"> the purpose of "coverage" is to be able to localize problems to a "covered thing" </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112089948930183338">May 27, 2023 11:48 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">this is certainly not the vibe I've gotten from the web dev world btw</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112090001291890688" class="chatlog__message-container" data-message-id="1112090001291890688"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112089948930183338')"> this is certainly not the vibe I've gotten from the web dev world btw </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112090001291890688">May 27, 2023 11:48 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">indeed no, but just study it as a "user experience"</span></div></div></div></div><div id="chatlog__message-container-1112090045256568852" class="chatlog__message-container" data-message-id="1112090045256568852"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:48 AM">11:48</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">test fails, now what? well, you go look at what is covered by the test</span></div></div></div></div><div id="chatlog__message-container-1112090082963378428" class="chatlog__message-container" data-message-id="1112090082963378428"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:48 AM">11:48</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">it is useful as a way to point attention at problems</span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112090169865162762" class="chatlog__message-container" data-message-id="1112090169865162762"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112090169865162762">May 27, 2023 11:49 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">the actual acted-out behavior is that tests are ways to point programmers at problems</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="handmadeThumbsUp"><img class="chatlog__emoji chatlog__emoji--small" alt="handmadeThumbsUp" src="files/857042369106804766-32647.png" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div><div id="chatlog__message-container-1112090254883688448" class="chatlog__message-container" data-message-id="1112090254883688448"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 11:49 AM">11:49</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">and they are useful to the extent they point programmers reliably at small loci of problems</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112090136835002428" class="chatlog__message-container" data-message-id="1112090136835002428"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="demetrispanos#5588">demetrispanos</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112089030654439444')"> so now you want your "covered things" to be useful attentional units </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112090136835002428">May 27, 2023 11:49 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I tend to find that many of the bugs that come up are either caught by asserts (which direct attention directly), or present themselves as user-visible behaviour that I probably wouldn't have foreseen for a module-level test
(but I may just be trying to justify not doing the work to add these <img loading="lazy" class="chatlog__emoji " alt="😄" title="smile" src="files/1f604-BF863.svg"> )</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:49 AM">(edited)</span></div></div></div></div></div>

<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112090307559968869" class="chatlog__message-container" data-message-id="1112090307559968869"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112090307559968869">May 27, 2023 11:49 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">this is a much more pragmatic view imo than any philosophy of "correctness"</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112090275398025226" class="chatlog__message-container" data-message-id="1112090275398025226"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="azmr#4870">Andrew (azmr)</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112088993388052540')"> it seems like a more apt (but harder to pin-down) metric is "coverage of potential state-space", which can be approximated as "coverage of potential inputs". This, I think, captures why I lean on root-level end-to-end integration tests: "are we looking right for the common and known-awkward user inputs?".
These will occasionally miss some important edge-cases (e.g. ring structures wrapping with a particular offset) that unit tests can then precisely target. </span> <span class="chatlog__reply-edited-timestamp" title="05/27/2023 11:46 AM">(edited)</span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112090275398025226">May 27, 2023 11:49 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">One big value I get from testing is to flip my mindset from code-centered to input-centered. It's a little like translating from time domain to frequency domain in Fourier analysis.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>

<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112090765175296112" class="chatlog__message-container" data-message-id="1112090765175296112"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112090765175296112">May 27, 2023 11:51 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">What would be an example of testing with an "input-centered" mindset vs. a "code-centered" mindset?</span></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112090369874735155" class="chatlog__message-container" data-message-id="1112090369874735155"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="Phil H#5516">Phil H</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112089194446192700')"> Yeah we had weekly review of the health of the services. The focus would be around post-mortems for any severe incidents that impacted users. We would also look at peak server load to see if we needed to expand and other "key performance metrics" such as latency to see if there were any regressions. For any post-mortems there was usually a chain of things that went wrong for it to get to that point and we assign urgent tasks to cover the holes in our process and testing. It could be anywhere from writing tests, adding live monitoring for something that went un-noticed, adding a new manual verification step to our build verifications etc </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112090369874735155">May 27, 2023 11:50 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">i think it's worth explicitly mentioning quickly that there's a difference between building out and testing a metric as a programmer, trying to engineer well-designed software, and as an employee at the company, using the metric as a Key Performance Indicator with a particular threshold to be met. The latter case won't always line up with the former case!</span></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112090931282317502" class="chatlog__message-container" data-message-id="1112090931282317502"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112090931282317502">May 27, 2023 11:52 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Well the KPIs were more about, "is the user's experience degrading" not "is this up to the company's standard". Having systems in place to measure that is invaluable.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="fire"><img class="chatlog__emoji chatlog__emoji--small" alt="🔥" src="files/1f525-8FE4F.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112091123532439583" class="chatlog__message-container" data-message-id="1112091123532439583"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112091123532439583">May 27, 2023 11:53 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">right, you have to be careful about how indirect your proxies are from the things you actually care about!</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112091568770383902" class="chatlog__message-container" data-message-id="1112091568770383902"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(250,186,71)" title="bvisness#0001">bvisness</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112090765175296112')"> What would be an example of testing with an "input-centered" mindset vs. a "code-centered" mindset? </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112091568770383902">May 27, 2023 11:54 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">In a code-centered mindset your answer to "when am I done writing tests" might be "when all lines are covered by tests". And if someone points out that there can still be bugs in different paths through the code, you tend to throw up your hands and say "larger space than atoms in the universe."

But with an input mindset you might notice that these few functions taken together are operating over a state space which has these 3 regimes separated by two borders. And you'll tend to naturally focus on testing points near the borders.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="handmadeThumbsUp"><img class="chatlog__emoji chatlog__emoji--small" alt="handmadeThumbsUp" src="files/857042369106804766-32647.png" loading="lazy"> <span class="chatlog__reaction-count">5</span></div><div class="chatlog__reaction" title="100"><img class="chatlog__emoji chatlog__emoji--small" alt="💯" src="files/1f4af-4CFF5.svg" loading="lazy"> <span class="chatlog__reaction-count">4</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112092343252832266" class="chatlog__message-container" data-message-id="1112092343252832266"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112092343252832266">May 27, 2023 11:57 AM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">That's an interesting contrast. I think gamedev demands a pretty input-centered mindset.
The whole point of a game is to generate this incredibly huge multidimensional state space of interesting things, and the player is continually moving around the space and prodding at the edges. You really can't often take a view of the software as lego blocks of code - you have to be almost hand-in-hand with the player in exploring every nook and facet of the state space. (Well, you're actually one step ahead, exploring it ahead of time).
You're <em>rarely</em> concerned with whether or not the software will crash! You're concerned with whether or not there's a compelling experience to be had in that corner of the fractal. If there isn't one right now, you make a choice as a designer to either gate off that corner of it, or to freshen it up with some potted plants.</span> <span class="chatlog__edited-timestamp" title="05/27/2023 11:58 AM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="map"><img class="chatlog__emoji chatlog__emoji--small" alt="🗺️" src="files/1f5fa-C949E.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112092892979277894" class="chatlog__message-container" data-message-id="1112092892979277894"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112092892979277894">May 27, 2023 12:00 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><div class="chatlog__markdown-quote"><div class="chatlog__markdown-quote-border"></div><div class="chatlog__markdown-quote-content"><span class="chatlog__markdown-mention" title="strager#1205">@strager</span> <a href="https://discord.com/channels/239737791225790464/708458209131757598/1112091206646759526" onclick="scrollToMessage(event, '1112091206646759526')">https://discord.com/channels/239737791225790464/708458209131757598/1112091206646759526</a>
This is why I like TDD. It makes me think about the feature..., not the production code. I naturally want to think about the code ('cus it's fun), so I need help thinking about the feature ('cus it's boring). 
</div></div>
There is something to this as a part of the creation process. I have definitely been guilty of having a vague idea of both the problem/input and solution/output and just immediately writing some code to try and bridge the gap without thinking through specifics.
Taking the time to simply specify what you actually want to happen - in prose, or diagrams, or tests (when they lend themselves to the problem domain)) - often makes the implementation process much more straightforward.

(But the tests here are just a formalised way of thinking through what you want to happen, running them &amp; confirming is mostly a secondary bonus)</span> <span class="chatlog__edited-timestamp" title="05/27/2023 12:01 PM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112092982770942109" class="chatlog__message-container" data-message-id="1112092982770942109"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112092343252832266')"> That's an interesting contrast. I think gamedev demands a pretty input-centered mindset.
The whole point of a game is to generate this incredibly huge multidimensional state space of interesting things, and the player is continually moving around the space and prodding at the edges. You really can't often take a view of the software as lego blocks of code - you have to be almost hand-in-hand with the player in exploring every nook and facet of the state space. (Well, you're actually one step ahead, exploring it ahead of time).
You're <em>rarely</em> concerned with whether or not the software will crash! You're concerned with whether or not there's a compelling experience to be had in that corner of the fractal. If there isn't one right now, you make a choice as a designer to either gate off that corner of it, or to freshen it up with some potted plants. </span> <span class="chatlog__reply-edited-timestamp" title="05/27/2023 11:58 AM">(edited)</span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112092982770942109">May 27, 2023 12:00 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Certainly a goal to aim for. I wonder how a long-lived game platform like say Eve Online or Minecraft deals with maintaining global coherence over a humongous state space over feature development spanning years.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112093161209221141" class="chatlog__message-container" data-message-id="1112093161209221141"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112093161209221141">May 27, 2023 12:01 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><div class="chatlog__markdown-quote"><div class="chatlog__markdown-quote-border"></div><div class="chatlog__markdown-quote-content">Minecraft
</div></div>slowly and painfully and often badly</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="joy"><img class="chatlog__emoji chatlog__emoji--small" alt="😂" src="files/1f602-168C5.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112093172949057537" class="chatlog__message-container" data-message-id="1112093172949057537"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:01 PM">12:01</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">as far as I can tell</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112093250233307290" class="chatlog__message-container" data-message-id="1112093250233307290"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112093250233307290">May 27, 2023 12:01 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">My hazy impression as well, but I don't play games much.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112093455607398430" class="chatlog__message-container" data-message-id="1112093455607398430"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408">Kartik Agaram</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112091568770383902')"> In a code-centered mindset your answer to "when am I done writing tests" might be "when all lines are covered by tests". And if someone points out that there can still be bugs in different paths through the code, you tend to throw up your hands and say "larger space than atoms in the universe."

But with an input mindset you might notice that these few functions taken together are operating over a state space which has these 3 regimes separated by two borders. And you'll tend to naturally focus on testing points near the borders. </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112093455607398430">May 27, 2023 12:02 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I agree with this sentiment. I'd be curious to hear how you think about or handle testing for a wide array of inputs. It has more often been the unexpected inputs that I don't account for in testing that come back to bite me. Sounds like <span class="chatlog__markdown-mention" title="azmr#4870">@Andrew (azmr)</span> has started to think about this with Whitebox fuzzing and generating complex inputs too?</span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112094164465746063" class="chatlog__message-container" data-message-id="1112094164465746063"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112094164465746063">May 27, 2023 12:05 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I like, I think it was <span class="chatlog__markdown-mention" title="philliptrudeau#8865">@Phillip Trudeau</span>'s example above, of just testing a function on all 32-bit ints. That sort of thing is possible if you can find an encoding that can traverse a space. Not always possible, but at least it's good to be thinking in this direction in case a solution occurs to you. Wouldn't happen if you always think in code-space.

<span class="chatlog__markdown-mention" title="demetrispanos#5588">@demetrispanos</span> also mentioned statistical tests above.</span></div></div></div></div></div><div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112096788669792387" class="chatlog__message-container" data-message-id="1112096788669792387"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112096788669792387">May 27, 2023 12:15 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I'm not sure I have any solutions for <em>unexpected</em> complex inputs before seeing them in the wild beyond regular fuzzing. It's much easier to capture than an existing rats-nest of pointers than to generate one, particularly one with semantics that the programmer knows but hasn't encoded in e.g. the type system...
Property testing probably has a place here, both in its normal form as a way of asserting outputs of fuzzed inputs, and as a non-location-specific generalization of asserts that account for time, which we'll be looking into with WhiteBox
<a href="https://www.tedinski.com/2018/12/11/fuzzing-and-property-testing.html" target="_blank">https://www.tedinski.com/2018/12/11/fuzzing-and-property-testing.html</a></span> <span class="chatlog__edited-timestamp" title="05/27/2023 12:18 PM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="fire"><img class="chatlog__emoji chatlog__emoji--small" alt="🔥" src="files/1f525-8FE4F.svg" loading="lazy"> <span class="chatlog__reaction-count">3</span></div></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112093695093776424" class="chatlog__message-container" data-message-id="1112093695093776424"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112093695093776424">May 27, 2023 12:03 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I can certainly say that at Mozilla we lean very heavily on fuzzing because of the enormous size of the space</span></div></div></div></div><div id="chatlog__message-container-1112093795698356375" class="chatlog__message-container" data-message-id="1112093795698356375"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:03 PM">12:03</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">and we do multiple kinds of fuzzing, both generative and mutation-based</span></div></div></div></div><div id="chatlog__message-container-1112093865353162772" class="chatlog__message-container" data-message-id="1112093865353162772"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:03 PM">12:03</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">there is very, very little "unit testing" in the firefox codebase</span></div></div></div></div><div id="chatlog__message-container-1112093984643358850" class="chatlog__message-container" data-message-id="1112093984643358850"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:04 PM">12:04</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">but lots and lots and lots of integration tests, which are continuously mutated by fuzzers to try and explore more of the space from those starting points</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="heart"><img class="chatlog__emoji chatlog__emoji--small" alt="❤️" src="files/2764-A3D25.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112094396091994223" class="chatlog__message-container" data-message-id="1112094396091994223"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112094396091994223">May 27, 2023 12:06 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Fuzzing ‼️
During development of the netcode for Sir Happenlance I came across a wonderful utility called clumsy, which can intercept any localhost loopback packets and lets you delay them, jitter them, reorder them, drop 10% of them, etc.
Using this, while two instances of the game were running and connected to each other, immediately exposed behaviour about the netcode that I wouldn't have otherwise been easily able to set up (I'd need two people around the world to run the dev build and connect).</span> <span class="chatlog__edited-timestamp" title="05/27/2023 12:06 PM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="100"><img class="chatlog__emoji chatlog__emoji--small" alt="💯" src="files/1f4af-4CFF5.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112094448000712875" class="chatlog__message-container" data-message-id="1112094448000712875"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112094448000712875">May 27, 2023 12:06 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">yes re: statistical tests there is an entire separate problem of testing things that don't have deterministic answers (for example, ML classifications of inputs)</span> <span class="chatlog__edited-timestamp" title="05/27/2023 12:06 PM">(edited)</span></div></div></div></div><div id="chatlog__message-container-1112094639122554942" class="chatlog__message-container" data-message-id="1112094639122554942"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:07 PM">12:07</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">and also yes, fuzzing (broadly interpreted) is great</span></div></div></div></div><div id="chatlog__message-container-1112094687780667552" class="chatlog__message-container" data-message-id="1112094687780667552"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:07 PM">12:07</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">generally randomized input testing</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112094815870533672" class="chatlog__message-container" data-message-id="1112094815870533672"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112094815870533672">May 27, 2023 12:07 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The sense I'm getting is that we're all in broad agreement about practices. We just have different boundaries for the word "testing".</span></div></div></div></div><div id="chatlog__message-container-1112094914700914778" class="chatlog__message-container" data-message-id="1112094914700914778"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:08 PM">12:08</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I certainly wish I knew more about fuzzing.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112095099950735360" class="chatlog__message-container" data-message-id="1112095099950735360"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112095099950735360">May 27, 2023 12:08 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I often combine fuzzing and soaking, so subject some system to 24 hours worth of randomized input to see if it breaks</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112095139540762697" class="chatlog__message-container" data-message-id="1112095139540762697"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112095139540762697">May 27, 2023 12:09 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">where there's agreement on practices, though, I hope to distill some of that for readers</span></div></div></div></div><div id="chatlog__message-container-1112095184700850216" class="chatlog__message-container" data-message-id="1112095184700850216"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:09 PM">12:09</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">especially as we are now past the two-hour mark already (?!)</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112095433469198399" class="chatlog__message-container" data-message-id="1112095433469198399"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112095433469198399">May 27, 2023 12:10 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve"><span class="chatlog__markdown-mention" title="Vegard#4618">@Vegard</span> has done a lot of great work on fuzzing, AFAIK - e.g. <a href="http://www.vegardno.net/2018/06/compiler-fuzzing.html" target="_blank">http://www.vegardno.net/2018/06/compiler-fuzzing.html</a></span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="heart"><img class="chatlog__emoji chatlog__emoji--small" alt="❤️" src="files/2764-A3D25.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112096022517256232" class="chatlog__message-container" data-message-id="1112096022517256232"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112096022517256232">May 27, 2023 12:12 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">So to try to wrap this up, what takeaways do you all have from this discussion?</span></div></div></div></div><div id="chatlog__message-container-1112096215648194662" class="chatlog__message-container" data-message-id="1112096215648194662"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:13 PM">12:13</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The things that have really stuck out to me are Demetri's point about "coverage" and the ultimate purpose of tests being to point programmers to problems, and Kartik's framing of "code-centered testing" and the problems it causes</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112096497719328878" class="chatlog__message-container" data-message-id="1112096497719328878"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112096497719328878">May 27, 2023 12:14 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I'd add that there is likely a judgement-free personality difference in that some people value designing around testing in its own right whereas others (like myself) see it purely transactionally</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div><div class="chatlog__reaction" title="heart"><img class="chatlog__emoji chatlog__emoji--small" alt="❤️" src="files/2764-A3D25.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112096543042969700" class="chatlog__message-container" data-message-id="1112096543042969700"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:14 PM">12:14</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">and this is fine, people should do what fits their minds</span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112096625708519564" class="chatlog__message-container" data-message-id="1112096625708519564"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112096625708519564">May 27, 2023 12:14 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">One of my takeaways is that I've got to figure out a mode of engaging with testing where I can be more systematic about it without reducing my agility</span></div></div></div></div><div id="chatlog__message-container-1112096782504177684" class="chatlog__message-container" data-message-id="1112096782504177684"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:15 PM">12:15</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">My brain is input-centered and it could do to add a sprinkling of code-centered</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="exploding_head"><img class="chatlog__emoji chatlog__emoji--small" alt="🤯" src="files/1f92f-FF4E0.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112097124771954788" class="chatlog__message-container" data-message-id="1112097124771954788"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112097124771954788">May 27, 2023 12:16 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I have some further ideas inspired by this conversation about test granularity too, although we didn't really discuss it much</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thinking"><img class="chatlog__emoji chatlog__emoji--small" alt="🤔" src="files/1f914-15707.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112097395031941160" class="chatlog__message-container" data-message-id="1112097395031941160"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112097395031941160">May 27, 2023 12:18 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I think I'll continue to be just a casual partaker in testing. When friends come over, etc. I'm wary of becoming a high functioning day-tester</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="laughing"><img class="chatlog__emoji chatlog__emoji--small" alt="😆" src="files/1f606-BE94E.svg" loading="lazy"> <span class="chatlog__reaction-count">5</span></div><div class="chatlog__reaction" title="kekw"><img class="chatlog__emoji chatlog__emoji--small" alt="kekw" src="files/865446640604938260-B3559.png" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112097485809254491" class="chatlog__message-container" data-message-id="1112097485809254491"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/8f4095ffc844c8dc8c175a55f48869c6-1728A.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Phil H#5516" data-user-id="277696761122783232">Phil H</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112097485809254491">May 27, 2023 12:18 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">One thing I'd like to add as we wrap up is that bugs are inevitable. Having a good strategy and system in place to minimize the effect of them when shipping code is just as important as testing, whether that's being able to rollback or hotfix it.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="fax"><img class="chatlog__emoji chatlog__emoji--small" alt="📠" src="files/1f4e0-936A5.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112097653854052403" class="chatlog__message-container" data-message-id="1112097653854052403"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/89747c254631bcfd05c5c3e7141389aa-A1FB9.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="demetrispanos#5588" data-user-id="239823533989953546">demetrispanos</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112097653854052403">May 27, 2023 12:19 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">one thing we didn't discuss is that I also think about testing in terms of consequences/responsibility in the real world, so for example totally separate from software concerns I want testing for things that (for example) push data to customers</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">4</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112097697571291186" class="chatlog__message-container" data-message-id="1112097697571291186"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112097697571291186">May 27, 2023 12:19 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">That's a good point. Every software has bugs, it's just a question of which stage of the pipeline you've evaluated is best for catching them - and that might be after the come out the pipe!</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112098015637938287" class="chatlog__message-container" data-message-id="1112098015637938287"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112098015637938287">May 27, 2023 12:20 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">The ideals to strive for are:
* small codebase
* lots of different ways to slice and dice parts of it to execute.
* some amount of cross-checking somehow. Execute something 2 ways, make sure they agree. This is a good lens for debates about types vs tests, in my experience. That's a rabbithole for another day.

Tests nudge me toward all 3 tendencies.</span> <span class="chatlog__edited-timestamp" title="05/27/2023 12:21 PM">(edited)</span></div></div></div></div></div>
<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112098289421144124" class="chatlog__message-container" data-message-id="1112098289421144124"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112098289421144124">May 27, 2023 12:21 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Oh, and bump fuzzing in my todo list.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="100"><img class="chatlog__emoji chatlog__emoji--small" alt="💯" src="files/1f4af-4CFF5.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112098066464505898" class="chatlog__message-container" data-message-id="1112098066464505898"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__reply-symbol"></div><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__reply"><img class="chatlog__reply-avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"><div class="chatlog__reply-author" style="color:rgb(22,235,183)" title="philliptrudeau#8865">Phillip Trudeau</div><div class="chatlog__reply-content"><span class="chatlog__reply-link" onclick="scrollToMessage(event,'1112097697571291186')"> That's a good point. Every software has bugs, it's just a question of which stage of the pipeline you've evaluated is best for catching them - and that might be after the come out the pipe! </span></div></div><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112098066464505898">May 27, 2023 12:20 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">That's related to the granularity stuff I was pondering. Might throw some ideas out there in <span class="chatlog__markdown-mention">#fishbowl-audience</span> afterward.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="fire"><img class="chatlog__emoji chatlog__emoji--small" alt="🔥" src="files/1f525-8FE4F.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>

<div class="chatlog__message-group" style=""><div id="chatlog__message-container-1112099795876720700" class="chatlog__message-container" data-message-id="1112099795876720700"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/c72723180bf059e8dbc35e6eaf9e0fa1-1FB1F.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="azmr#4870" data-user-id="240986595749134336">Andrew (azmr)</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112099795876720700">May 27, 2023 12:27 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Perhaps obvious, but it seems like you need to make sure that you're not adding tests mindlessly. The purpose(s) you have for it may influence the type of tests you (don't) write:
- helping you think through the problem spec
- checking correspondence to pre-existing spec
- acting as example usage code
- preventing regressions for previously seen issues
- recreating a particular scenario to inspect repeatedly
- adding friction intentionally(!) to ensure focus from other devs/future you
- making sure that potential inputs don't put your program in an invalid state (i.e. sending out example runs through a minefield of assertions you've laid for them)
- ensuring that the thing you're sending to customers does what you say it will</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="point_up"><img class="chatlog__emoji chatlog__emoji--small" alt="☝️" src="files/261d-23B11.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="100"><img class="chatlog__emoji chatlog__emoji--small" alt="💯" src="files/1f4af-4CFF5.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">4</span></div><div class="chatlog__reaction" title="thinking"><img class="chatlog__emoji chatlog__emoji--small" alt="🤔" src="files/1f914-15707.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112100139578961951" class="chatlog__message-container" data-message-id="1112100139578961951"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:28 PM">12:28</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">This then has to be considered in the context of the type of code you're writing <a href="https://discord.com/channels/239737791225790464/1112062893517717514/1112071588343459900" target="_blank">https://discord.com/channels/239737791225790464/1112062893517717514/1112071588343459900</a></span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="smart"><img class="chatlog__emoji chatlog__emoji--small" alt="smart" src="files/867119742622892042-C589A.png" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class=chatlog__message-group><div id=chatlog__message-container-1112104991478517821 class=chatlog__message-container data-message-id=1112104991478517821><div class=chatlog__message><div class=chatlog__message-aside><img class=chatlog__avatar src=files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png alt=Avatar loading=lazy></div><div class=chatlog__message-primary><div class=chatlog__header><span class=chatlog__author style=color:rgb(22,235,183) title="Kartik Agaram#9408" data-user-id=692826981640831107>Kartik Agaram</span> <span class=chatlog__timestamp><a href=#chatlog__message-container-1112104991478517821>May 27, 2023 12:48 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class=chatlog__markdown-preserve>I didn&#39;t really address asserts at all. They feel complementary to tests:
* Tests: don&#39;t run in dev, allow you to assess portions of a program over time
* Assertions: can run in production (nobody ever enables <code class="chatlog__markdown-pre chatlog__markdown-pre--inline">NDEBUG</code>, right?), can only check what&#39;s in scope at a specific point in time.
    
Before I understood tests I often tried to periodically go into a mode where the program would perform sanity checks on some internal data structure.</span></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112098302603833404" class="chatlog__message-container" data-message-id="1112098302603833404"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112098302603833404">May 27, 2023 12:21 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I wish we could keep going but 2.5 hours is a long time to keep everyone! I feel like we've barely scratched the surface. I think we'll definitely have to have another fishbowl that digs deeper into some of the topics you all brought up today.</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="thumbsup"><img class="chatlog__emoji chatlog__emoji--small" alt="👍" src="files/1f44d-27259.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div><div id="chatlog__message-container-1112098595534024845" class="chatlog__message-container" data-message-id="1112098595534024845"><div class="chatlog__message"><div class="chatlog__message-aside"><div class="chatlog__short-timestamp" title="05/27/2023 12:22 PM">12:22</div></div><div class="chatlog__message-primary"><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Thanks all for being here! If anyone would like to keep going, you can continue in <span class="chatlog__markdown-mention">#fishbowl-audience</span> - otherwise, enjoy the rest of your Saturday (and your Memorial Day weekend for those in the US!)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="heart"><img class="chatlog__emoji chatlog__emoji--small" alt="❤️" src="files/2764-A3D25.svg" loading="lazy"> <span class="chatlog__reaction-count">2</span></div></div></div></div></div></div><div class="chatlog__message-group"><div id="chatlog__message-container-1112098784701325363" class="chatlog__message-container" data-message-id="1112098784701325363"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112098784701325363">May 27, 2023 12:23 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Thanks so much, everyone! ❤️</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="heart"><img class="chatlog__emoji chatlog__emoji--small" alt="❤️" src="files/2764-A3D25.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112099029938094145" class="chatlog__message-container" data-message-id="1112099029938094145"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/3bf987dc7a33d0b2f989878615544d6a-D4B96.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="Kartik Agaram#9408" data-user-id="692826981640831107">Kartik Agaram</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112099029938094145">May 27, 2023 12:24 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">I said in my last fishbowl that I no longer understand why people attend conferences. This time I am glad to report I have attended 0 conferences in the interim <img loading="lazy" class="chatlog__emoji " alt="😛" title="stuck_out_tongue" src="files/1f61b-EC4B5.svg"></span> <span class="chatlog__edited-timestamp" title="05/27/2023 12:24 PM">(edited)</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="laughing"><img class="chatlog__emoji chatlog__emoji--small" alt="😆" src="files/1f606-BE94E.svg" loading="lazy"> <span class="chatlog__reaction-count">3</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112099132086177892" class="chatlog__message-container" data-message-id="1112099132086177892"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112099132086177892">May 27, 2023 12:24 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">we'll drag you to Handmade Seattle one of these years...</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="heart"><img class="chatlog__emoji chatlog__emoji--small" alt="❤️" src="files/2764-A3D25.svg" loading="lazy"> <span class="chatlog__reaction-count">1</span></div></div></div></div></div></div>
<div class="chatlog__message-group"><div id="chatlog__message-container-1112099190047244330" class="chatlog__message-container" data-message-id="1112099190047244330"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/06ae5aa4cd2d290bf1a433d6cad34b0f-50E28.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(22,235,183)" title="philliptrudeau#8865" data-user-id="335466817910996992">Phillip Trudeau</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112099190047244330">May 27, 2023 12:25 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">Realtime fishbowling sounds like it'd be super productive!</span></div></div></div></div></div>

<div class="chatlog__message-group"><div id="chatlog__message-container-1112100431846453300" class="chatlog__message-container" data-message-id="1112100431846453300"><div class="chatlog__message"><div class="chatlog__message-aside"><img class="chatlog__avatar" src="files/1963eacbf364164efce1c597dc66aeab-E6901.png" alt="Avatar" loading="lazy"></div><div class="chatlog__message-primary"><div class="chatlog__header"><span class="chatlog__author" style="color:rgb(250,186,71)" title="bvisness#0001" data-user-id="132715550571888640">bvisness</span> <span class="chatlog__timestamp"><a href="#chatlog__message-container-1112100431846453300">May 27, 2023 12:30 PM</a></span></div><div class="chatlog__content chatlog__markdown"><span class="chatlog__markdown-preserve">ok now we're done for real!</span></div><div class="chatlog__reactions"><div class="chatlog__reaction" title="fire"><img class="chatlog__emoji chatlog__emoji--small" alt="🔥" src="files/1f525-8FE4F.svg" loading="lazy"> <span class="chatlog__reaction-count">4</span></div></div></div></div></div></div>



</div>