Key Contributions

• Defining and Characterizing Test Flimsiness: We introduce and formally characterize the concept of test flimsiness, where standard mutation operators applied to the code under test cause previously stable test cases to start failing intermittently. This is the first study to systematically investigate this phenomenon.

• Large-Scale Empirical Study: We conducted an extensive empirical evaluation across 28 real-world Python projects, spanning over half a million test suite executions. Using rigorous statistical analysis with false discovery rate controls, we reliably identified transitions from stability to flakiness triggered by mutations.

• A New Lens for Flaky Test Detection: Prior work perturbed the execution environment or injected flakiness into test code. We advance these efforts by perturbing a third major source of flakiness: the code under test. Our mutation-based rerunning strategy detects far more flaky tests than standard rerunning alone.

• Comprehensive Public Dataset: We release a rich dataset comprising over half a million test suite runs and 30 million mutant-test pairs, annotated with test outcome reports and coverage data, in the paper’s replication package.

Empirical Results

Our study revealed several noteworthy findings about test flimsiness:

• Flimsiness is prevalent: It occurs in 15 out of 28 projects (54%), with 0.7% of all mutants inducing flakiness in at least one test case. While the percentage may seem small, we argue that the implications for mutant-driven techniques are significant.

• Mutation operators differ in their tendency to induce flakiness: For example, mutants produced by the ReplaceUnaryOperator have over 3 times higher odds of inducing flakiness compared to other operators. Interestingly, operators that are better at producing killable mutants also tend to be better at inducing flakiness.

• Mutation-based rerunning detects far more flaky tests: The mutation-based strategy detected a median of 740 flaky tests per project compared to just 163 for standard rerunning. The flaky tests found by the mutation-based strategy are especially unlikely to be detected by the standard strategy, reinforcing flimsiness as a distinct form of flakiness worthy of further study in future research.

• Unstable coverage does not explain flimsiness: Whether a test case non-deterministically covered the mutated line prior to mutation is a poor predictor of whether the mutant will induce flakiness. This suggests that the causes of flimsiness are more nuanced than simply manifesting unstable coverage.

Future Work

The paper opens several avenues for future investigation. My co-authors and I plan to study how flimsiness impacts specific mutant-driven techniques such as fault localization and regression testing. We also intend to survey developers to understand how they perceive flimsiness and whether they encounter it during routine use of mutation testing tools. Finally, we aim to investigate the root causes of flimsiness more systematically, potentially involving project developers in the analysis.

Return to Blog Post Listing

Insights

I’m thankful for this fun-loving and engaging discussion with both Jennings Anderson and Amy Rose. Before you listen to the full episode, you can preview some of its insights:

Why is normalizing geospatial data to a single schema so important?

Amy explains the challenge that Overture Maps solves for developers:

“Imagine you’re trying to build an application that uses map data. You’re pulling all this information from different sources if each source has its own unique way of describing things. So let’s say some data refers to roads as quote unquote streets and others as highways. Or if you have in some datasets, the height is measured in meters and others in feet just because of cultural differences. It’s a total nightmare to get them to all work together, right?”

What is the Global Entity Reference System (GERS) and why does it matter?

Jennings describes the innovation behind stable identifiers in Overture Maps:

“The idea of GERS is to say, okay, well there might be different representations of each of these buildings, but each building is in fact its own building entity, the building-ness of the building, so to say. So we will define a single entity for a building, give that a unique identifier.”

How does DuckDB enable developers to query Overture Maps data efficiently?

Jennings explains the cloud-native workflow that Overture Maps makes possible:

“Our data is all released in GeoParquet format, which is a columnar format that’s optimized for the cloud native environment here. And DuckDB is a fantastic open-source query engine that allows us to investigate that data in situ in the cloud. … However, with DuckDB and with Geo Parquet, what that enables us is the ability to actually query the data in place.”

How is the relationship between open-source software and open data different?

Amy highlights a distinction that is often overlooked by software engineers:

“Open-source software and open-source data you might want to lump them together because it’s all open source, but they’re very different things. … How you build open-source software is much clearer. There’s a lot of tools out there already to very cleanly execute on these types of collaborations, particularly in big projects. And code is pretty easy to think about how you would release and continue to maintain and update. Data’s a very different animal, not the least of which is because it can get really big.”

Why is Overture Maps part of the Linux Foundation?

Amy explains the governance model that guides the project:

“As part of the Linux Foundation, Overture operates as a nonprofit technology consortium. So the idea of operating under that structure is that it provides a neutral, kind of vendor agnostic, home for the project. It gives us a much better way to remain open and truly collaborative.”

Listen

If you’re interested in learning more about Overture Maps and open geospatial data, I highly recommend listening to my interview with Jennings Anderson and Amy Rose on Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or you can listen to it with this handy podcast player.

Return to Blog Post Listing

Insights

I’m thankful for this action-packed and informative discussion with Kacper Łukawski! Before you listen to the full episode, please checkout some of its insights:

What is a vector database and how does similarity search work?

“It’s a service that can efficiently store and handle high dimensional vectors for the proposals of similarity search and similarity of these vectors is defined by the closeness of the vectors in that space. So vector databases are built to make that process efficient.”

What are vector embeddings and why are they important for semantic search?

“Vector embeddings are just numerical representations of the input data and the main idea is that they keep the semantic meaning of the input data that was used to generate them. And if we have two different vectors which are similar in some way, then we assume that the objects that are used to generate them are also similar in their nature.”

Why did the Qdrant team choose Rust as the implementation language?

“Definitely the biggest factor behind choosing Rust is its safety. And we can achieve almost similar performance like C or C++ sometimes even better. While keeping this language safety and this strong type system that Rust provides is very helpful in preventing us from making some mistakes in a highly concurrent system. So reading or writing some value from multiple threats concurrently because that’s ultimately what you can expect from a search engine.”

What metrics does Qdrant use when benchmarking performance?

“Depending on a specific use case you have or maybe some budget constraints, you might prefer to optimize for a particular metric from those four. But we measure all of them and report them in our benchmarks just so you can have understanding of what you can expect in a very specific setup. For example, low latency might be important if your users expect immediate response, and we measure an average latency P95 and P99 so we can see like what the majority of users can expect from the system and how fast it is going to be.”

What is the future of vector databases in the AI landscape?

“The main problem of AI or LLMs … even the latest LLM can suffer from knowledge cutoff because they were trained on some specific data sets and definitely don’t know that most recent news and none of them could have been trained on your own data. So definitely some sort of retrieval is needed and vector databases will definitely serve that functionality.”

Listen

If you’re interested in learning more about Qdrant and vector similarity search, I highly recommend listening to my interview with Kacper Łukawski on Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or you can listen to it with this handy podcast player.

Return to Blog Post Listing

Insights

I’m thankful for this thought-provoking discussion with Samuel Colvin! Before you listen to the full episode, please checkout some of its insights:

What is Pydantic and how does it help with data validation?

“So Pydantic, to all intents and purposes, is a data validation library for Python. But it does a bunch more stuff above and beyond simple data validation — like controlling serialization, coercion when you’re validating and generating JSON schema.”

What were the motivations for rewriting Pydantic V2 in Rust?

“Initially the internals of Pydantic were all Python. I wasn’t particularly proud of the internals. And so at the beginning of 2022, I started working full-time on Pydantic, rebuilding the core in Rust. There’s an amazing library called Py03 that allows you to write Python extensions in Rust, which gives you an enormous performance improvement. But it’s actually, it goes beyond simply the performance. It allows you to write fundamentally more robust and scalable and maintainable code when you’re building these big, performance- critical complex applications.”

How does Pydantic AI help developers interact with large language models (LLMs)?

“So the number one piece of value I think that Pydantic AI brings is this model agnosticism. You can switch in and out of OpenAI, Anthropic, Gemini, Grok, DeepSeek, et cetera, et cetera, Mistral, Cohere, with one line of code.”

What is Logfire and how does it help with observability?

“Logfire is an observability platform. We do logs, traces and metrics that let you understand and monitor your application. It is very useful when you’re building applications in Python in general, but particularly AI applications.”

Listen

If you’re interested in learning more about Pydantic and its ecosystem, I highly recommend listening to my interview with Samuel Colvin on Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or you can listen to it with this handy podcast player.

Return to Blog Post Listing

Key Contributions

This paper’s research offers a comprehensive empirical comparison of three different Oracle Gap Calculation Approaches (OGCAs) and the gaps that they produce:

• Checked Coverage using a Dynamic Slicer (CCDS): This technique uses dynamic slicing to identify statements that influence the outcome of a test assertion.
• Checked Coverage using an Observational Slicer (CCOS): This approach repeatedly deletes lines of code to see if the program’s behavior changes.
• Pseudo-Tested Statement Identification (PTSI): This method identifies the pseudo-tested statements that can be removed without causing any test to fail.

This paper’s study gives a quantitative and qualitative analysis of these techniques, helping developers choose the most suitable approach for their needs. Read on to learn more!

Empirical Results

We conducted an empirical study on 30 Java classes from six open-source projects. This paper identifies several interesting results such as the following:

• PTSI is the most efficient and effective OGCA: This method consistently identified the oracle gaps with the lowest mutation scores, indicating that it is the best at pinpointing areas where the test suite’s fault detection is weak.
• PTSI is the fastest approach: Using pseudo-testedness to identify program statements in the oracle gap has the best performance, making it a practical choice for developers who need to quickly assess the strength of their test suites.
• Distinct oracle gaps: The three OGCAs created oracle gaps that were largely distinct, suggesting that they uncover different types of testing weaknesses.

Future

While our study focused on programs implemented in Java, the concept of oracle gaps is applicable to a wide range of programming languages. Future work will involve expanding our analysis to other languages and exploring how these techniques can be integrated into the software development lifecycle to provide developers with real-time feedback on their tests. Ultimately, we hope that the current and future versions of the GapGrep tool can offer developers an automated approach to identifying how their test cases fall short.

Return to Blog Post Listing

Summary of Key Results

• The Prevalence of Systemic Flakiness: Our analysis of 810 flaky tests across 24 Java projects revealed that 75% of flaky tests belong to clusters with an average of 13.5 tests per cluster. This means that when one flaky test fails, it’s likely that 12 or 13 other test cases will also fail at the same time due to shared root causes.

• Predicting Systemic Flakiness with Machine Learning: Rather than requiring thousands of test suite runs to identify which tests fail together, we found that machine learning models can predict systemic flakiness using simple static analysis. Extra trees models achieved the best performance with 74% accuracy, and surprisingly, the structure of test names (like package and class hierarchy) was more predictive than analyzing the actual source code.

• Root Causes of Systemic Flakiness: Through manual inspection of error messages and stack traces, we discovered that systemic flakiness is primarily caused by networking issues (e.g., domain name service failures or connection timeouts) and external dependencies (e.g., unstable services or library version conflicts). This differs significantly from individual flaky tests, which are usually caused by concurrency or timing issues.

Future Research Suggestions

This paper’s results suggest several exciting avenues for future investigation:

• Expanding Beyond Java: The current study focused on Java projects, but systemic flakiness likely occurs across all programming languages and testing frameworks. Research should examine whether similar clustering patterns exist in other ecosystems.

• Automated Root Cause Analysis: While our manual inspection revealed important patterns, the process was time-consuming and sometimes inconclusive. Developing artificial intelligence (AI)-powered techniques to automatically analyze test code and stack traces could significantly accelerate the identification of systemic flakiness causes.

• Integration with CI/CD Tools: Current flaky test detection tools normally focus on individual tests without considering failure co-occurrence. Research should explore how to integrate systemic flakiness prediction into existing continuous integration (CI) and continuous delivery (CD) pipelines on platforms like GitHub Actions.

• Developer Studies: Our research was based on quantitative analysis, but we need to understand how developers currently perceive and handle systemic flakiness. Surveys and interviews could reveal whether practitioners are already aware of this phenomenon.

• Realistic Testing Impact: Previous studies evaluated how flaky tests affect fault localization and mutation testing using simulated individual failures. These assessments should be revisited using realistic models that account for clustered failures.

• Benchmark Datasets: The research community needs datasets that explicitly capture failure co-occurrence patterns and the phenomenon of systemic flakiness. These benchmarks would enable comparative evaluation of systemic flakiness detection tools.

Return to Blog Post Listing

Insights

I’m thankful that Will took the time for this interview! Here are some key insights from it:

What makes text-based user interfaces compelling for developers?

“They tend to be very responsive and very quick and very snappy and there’s less contact switching. I mean I used to be a web developer, I kind of use the web a lot like most people do, but there’s quite a lot of contexts switching when you go between web application, browser tab, etc. But the benefits of running in the terminal is that it stays in your own flow. So it’s like everything’s in one place and it’s snappy and responsive and it comes up when you need it. It’s very quick to interact with and then it goes away as quickly when you don’t need it.”

How does bringing web technologies to the terminal improve development?

“So I come from the web world and the great thing about the web is that building user interfaces has been iterated on very rapidly for the last 10, 20 years. People have been trying new ways, new interfaces, refining it and creating new technologies to create these user interfaces. But none of that made it to the terminal. The terminal was considered a bit archaic and people used it for more simplistic outputs. So I took web technologies and essentially ported what I think is the best of them to the terminal.”

What performance strategies are crucial for smooth terminal interfaces?

“You consider the terminal to be a matrix of characters. That’s essentially all, it’s that composes all the elements that run inside the terminal and if you change it, you can build up a new frame. If you were to clear the screen, which might be obvious and then write your new screen on top of it, you get flicker because the terminal was never really designed. In a sense it was, but it’s not good at displaying user interfaces. That’s not really its primary purpose even though it can, the way around that is instead of clearing the screen, you just write the new content on top of it and do you need to change the bits of the screen which have logically changed.”

How does caching with immutable objects improve performance?

“Immutability is the opposite of mutability. Mutability means the ability to change, mutate your data. If an object isn’t mutable, it’s immutable and it can’t change and it might seem like a restriction, like you’ve got a data, you’ve got an object, you can’t change it. It feels like a restriction but it is massively beneficial. It means that when you change it, you create a new object rather than modifying the existing one. And when you do that, caching works a lot better.”

Listen

If you’re interested in learning more about Rich, Textual, and building text-based user interfaces in Python, I highly recommend listening to my interview with Will McGugan about text-based user interfaces on Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or you can listen to it with this handy podcast player.

Return to Blog Post Listing

Insights

I’m thankful for this thought-provoking interview with Eran Yahav! Here are some of its insights:

How does Tabnine’s enterprise context engine make AI assistants more effective?

“From the LLM to an actual use of software engineer, there is a hole that has to happen in the middle, right? Part of that is the context that I mentioned earlier. So Tabnine has what we call the enterprise context engine, which is something that basically knows how to draw relevant context from all code sources of information and non-code sources of information in the organization to make sure that again, Tabnine operates like an onboarded employee of the organization and not as a foreign engineer in the org. So Tabnine knows everything about the org, and which informs the code that it generates, how it reviews code, etc.”

What is the vision for AI assistance across the software development lifecycle?

“Tabnine provides assistance across the entire SDLC. It helps you write code with code completions that are very advanced. It has a chat interface that allows you to create new code, review code, refactor code, translate between languages, generate tests as you mentioned initially, document code, explain it. So it basically helps you do anything that you have to do as a software engineer and the vision is really to provide agents that help you do anything and everything a software engineer does just faster and with a higher quality.”

How should software engineers think about the future of AI in programming?

“I think really the future is every engineer is really a team lead and that team lead is leading a team of AI engineers. They could be specialized in different domains, the AI engineers, they could have varying expertise, but I think every engineer would have to start thinking like an engineering manager in a sense, or at least like a team lead.”

What makes the difference between AI assistants and existing tools like ChatGPT?

“You can think about the LLM as a kind of an ignorant genius. It knows how to do many, many, many things, but not in in the relevant context in which I’m operating, right? And you can think about the best engineer in the organization as someone who really knows the nuts and bolts of how things operate, right? […] So you want really to bridge these two things, make the LLM as aware of the org as your best engineer and this is how you get like to 10x productivity of the AI engineer, right? To onboard the AI to your organization.”

Listen

If you’re interested in learning more about Tabnine and exploring the landscape of AI-powered software engineering tools, I highly recommend listening to my interview with Eran Yahav about Tabnine on Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or you can listen to it with this handy podcast player.

Return to Blog Post Listing

Key Contributions

• Nine Health Indicators: We identify a checklist of indicators that signal an unhealthy test suite, starting with flakiness but extending to low code coverage, pseudo-testedness, low mutation scores, long-running test suites, low test diversity, high brittleness, low realism, and high variability of indicator metrics.

• Trade-offs Between Indicators: We argue that some indicators are complementary while others are in tension. For instance, a test suite with fewer assertions may be less flaky but also more pseudo-tested, meaning it executes code without actually checking it. Pursuing a high mutation score might increase brittleness if tests become tightly coupled to implementation details rather than intended behavior.

• A Research Agenda: We outline seven challenges for the research community, ranging from identifying further indicators and quantifying trade-offs to building tooling that can give developers actionable recommendations for improving the overall health of their test suites.

Key Insights

Since this is a short position paper and manifesto, the contribution is conceptual rather than experimental. The paper synthesizes insights from across the software testing literature to make the case that indicators like pseudo-testedness, brittleness, and realism deserve the same level of research attention that flakiness has received. The accompanying resource page catalogs existing detection and improvement tools for each indicator, showing where tooling already exists and where gaps remain.

Future Work

The paper outlines several open challenges, including how to measure less well-understood indicators like test realism and brittleness, how to combine multiple indicators into a composite picture of test suite health, and how to study the evolution of test suite health over the lifecycle of a project. Ultimately, the goal is to move toward automated tools that not only diagnose problems but also recommend concrete actions for developers to improve their test suites.

Return to Blog Post Listing

Understanding XMT

An XMT tool automatically detects deficiencies in a test suite by deleting method bodies in covered code and observing whether the test suite detects their absence. If a method deletion goes unnoticed, then the method is labeled as “pseudo-tested,” meaning that the test suite calls the method but its result isn’t checked by the assertions, indicating a potential blind spot in the test suite. As an extension to XMT, our paper explores the use of the statement deletion mutation operator (SDL) to uncover pseudo-tested statements within methods.

Research Insights

Using the PseudoSweep tool (Maton, Kapfhammer, and McMinn 2024b) to perform mutation testing with the XMT and SDL operators on four Apache Commons projects and 23 projects randomly selected from the Maven Central Repository, this paper’s experiments find 722 different cases of pseudo-tested statements. Interestingly, the results show that 48% of pseudo-tested statements exist outside of pseudo-tested methods! This suggests that XMT alone is insufficient and should be combined with pseudo-tested statement evaluation to find testing oversights that a developer should tackle before using standard mutation testing.

Future Directions

While our research focused on Java programs, the concepts of XMT and pseudo-tested methods are broadly applicable. These techniques can be used in various programming languages to improve test suite effectiveness and ensure more robust software. We’re excited about the potential of combining XMT with other testing techniques to further enhance test suite robustness. Future work will involve extending our approach to different programming languages and exploring additional ways to detect and repair pseudo-tested methods.

Return to Blog Post Listing

Insights

I’m thankful that Itamar took the time for this interview! Here are some key insights from it:

What testing tasks does Qodo-Cover automatically perform?

“Qodo-Cover can try to generate all types of testing, but the sweet spot that we saw is mostly around component testing. If you provide an initial few tests that cover one or more components and you run Qodo-Cover, it’ll try to generate many more. It takes the first few tests that were given as part of the test suite and exploits these to inspire it to generate more.”

Why is code coverage still a valuable metric despite its limitations?

“Many would say that code coverage is a proxy metric. I think almost 100% people would say that. But I think that also some percentage of developers and managers would say that code coverage is actually a vanity metric. Practically what we’re trying to say is that did we check that the code work exactly as expected? Okay and now how do you check that?”

What is a flaky test case? How does Qodo-Cover handle flaky test cases?

“If you have a test and you’re running it five times and it doesn’t give you the same result each time, especially if sometimes it’s passed, sometimes it fails, that’s the most obvious flakiness, then it’s flaky. It means how can you trust that test if it passed four times the same test, you know, just run it five times and pass four times and fail one that’s a flaky test. LLMs, you can try to prompt it not to generate the flaky test. I think the prompting to start with will try to do that and because there are tests smells that can be smelled for example if there is an obvious API call an obvious one.”

What advice do you have for developers adopting AI-powered development tools?

“About dev tools, I think developers love writing code, they love building and now they can, you know, use code completion to do even a bit more, probably save time and go drink more coffee. I don’t think it’s really helpful for generating more features. But actually let’s think about the real bottlenecks. I don’t think it’s writing more lines of code. It’s about code quality, it’s about code reviewing, it’s about testing, etc. So I think that when you want to start integrating, you know, different AI dev tools to improve your productivity, think what are your ‘today bottlenecks’ and what are your ‘today problems’?”

Listen

If you’re interested in learning more about automated testing with generative AI and how tools like Qodo-Cover can enhance your development workflow, I highly recommend listening to Itamar Friedman on Automated Testing with Generative AI from Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or you can listen to it with this handy podcast player.

Return to Blog Post Listing

Insights

I’m thankful that Goran took the time for this interview! Here are some key insights from it:

What is mutation testing and how does it work?

“Mutation testing is just a step further in this to ensure that the code is not only covered with tests, but actually properly tested in terms of assertions. You could imagine that you can have a function that you call from a test and don’t assert anything. It can have 100% coverage, but it will never catch any real bugs. And mutation testing comes to help in terms of validating that these tests do something useful.”

Why was it initially hard to convince engineers at Google to use mutation testing?

“Mutation testing generates a huge amount of mutants and then shoves them down your throat. And when I started implementing this, what I did is I read a bunch of research papers. I was really excited. I started coding and in a few weeks I had something that worked originally for C++. And then I started generating these mutants. […] And my first run, I remember it took a few hours to run. I ran it on my on a small code base of my team. […] And when I when it finished, I looked at the file and could barely open it in Vim. It was huge! There were thousands, tens of thousands of mutants. And after three pages and two coffees, I was so tired that I just closed the file and forgot about it for a few weeks because there was a huge amount of mutants. And most of them were not things that I would actually be adding tests for!”

How did you know that it was the right time to introduce mutation testing at Google?

“I joined the team in Google Shopping. And as it used to be the case, everyone had a 20% project. And my manager said, oh, we started this mutation testing last hackathon. Maybe you can inherit it. And I’m like, oh, I have no idea what this is. I’ve never heard of this, but I will inherit it because I don’t really have a choice. So I started reading those papers and eventually rewrote all of their code because it was a hackathon project one week of like just try to get a proof of concept working. And then that it became my project over time!”

Listen

If you’re interested in learning more about mutation testing and software engineering practices at Google, I highly recommend listening to Goran Petrovic on Mutation Testing at Google from Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or you can listen to it with this handy podcast player.

Return to Blog Post Listing

Insights

I learned a lot from Jonathan during this interview. Here are some of his key insights:

What is automated refactoring and what are its benefits?

“Automated refactoring is writing software that rewrites other software. Its main benefits are to achieve some modernization in that software, to fix security vulnerabilities, and to ensure a consistent style or fix inconsistencies in codebases.”

How does OpenRewrite ensure the correctness of automated refactoring?

“The correctness really stems from some of the characteristics of that data that we mentioned earlier. It’s when a recipe is able to use the type attribution information that the compiler knows that this info method comes from a particular type that we’re able to make that change. So type attribution is one of the essential elements of correctness in a recipe.”

How does OpenRewrite handle the challenge of code quality issues?

“We’ve been taking those recommendations and then matching them to the recipe catalog, those 2,200 recipes that exist, pre-running them, and when they make change, then we have a signal that there is something to do on the codebase at large.”

Listen

In conclusion, if you’re interested in learning more about automated refactoring with OpenRewrite, I highly recommend that you listen to Jonathan Schneider on Automated Refactoring with OpenRewrite from Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or listen to it with this handy podcast player.

Return to Blog Post Listing

Insights

These three questions overview my discussion with Marcelo about FastAPI:

What is FastAPI and why is it important?

“FastAPI, it’s a framework that’s used to build APIs in Python. It leverages type hints and it’s very fun to develop with.”

How does FastAPI ensure high performance and developer productivity?

“So for me I always saw, I mean even if I’m the maintainer of Uvicorn, the server that makes is one of the reasons that FastAPI to be fast on my perspective, the fast is more about the user perspective. Because in any case you’re still working with Python and there are certain limitations but the idea is that since you’re leveraging the async world, you don’t need to use the threads. So you can have more tasks running in a single thread and then you have more capacity to run more tasks and that means that you can receive and process more requests and that makes the server faster I would say.”

How does FastAPI handle authentication and authorization?

“So FastAPI does not handle that for you. So you need to implement the layer yourself. What it does is you can define a way for how to interpret some data that comes from the request. So for example, you have HTTP basic class, which it’s used for FastAPI to get the authorization header and then get the username and the password and it provides that to your endpoint. So you have, you use that class just to kind of parse that data for you, but it does not handle that for you.”

Listen

In conclusion, if you’re interested in learning more about the design and implementation of FastAPI or approaches to building APIs in Python, I highly recommend that you listen to Marcelo Trylesinski on FastAPI from Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or listen to it with this handy podcast player.

Return to Blog Post Listing

Insights

Here are three takeaways from my interview with Wolf Vollprecht:

Why did Wolf choose the Rust programming language to build Pixi?

“It was not initially a completely obvious decision because I already had a lot of experience with C++ and Mamba and we could have stuck to the C++ ecosystem, but in hindsight I’m very happy that we chose Rust. It comes with this rich ecosystem; it helped us to use the same sort of underlying code in our website as well and our platform and we want to capitalize on that in the future much more by building awesome services.”

How does Pixi ensure reproducibility in Python projects?

“Reproducibility in the scientific world is also a big problem. And with Pixi we want to make sure that the scientist that uses Pixi can get the same results now and then also get the same results in 10 years if you run the same data pipelines and things like this. And we make sure of that by creating a very tight lock file that locks down packages down to the SHA hash.”

How does Pixi handle the challenge of cross-platform compatibility?

“So the majority of work was already done because, well we are piggybacking on the Conda ecosystem and Conda Forge specifically as a really large repository of existing packages. And so a lot of the nice tools that we want to use, or our users might want to use are already available. But then there are some more intricate challenges because Windows and Unix are usually quite different.”

Listen

In conclusion, if you’re interested in learning more about the challenges and benefits of implementing Python tooling in Rust, I highly recommend that you listen to Wolf Vollprecht on Python Tooling in Rust from Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or listen to it with this handy podcast player.

Return to Blog Post Listing

Key Contributions

Our paper makes the following key contributions to the field of flaky test detection:

• CANNIER Approach: A novel technique that uses machine learning models as a heuristic to significantly reduce the problem space for rerunning-based flaky test detection. For many projects, CANNIER leads to a dramatic reduction in the time cost of detection with only a minimal decrease in performance.

• Extensive Tooling: The lead author developed and released a comprehensive framework of automated tools to facilitate the replication of our results and to empower other researchers to build upon our work in hybrid flaky test detection.

• Comprehensive Empirical Evaluation: A study involving nearly 90,000 test cases from 30 Python projects that not only demonstrates the effectiveness of CANNIER but also reveals new insights into machine learning-based flaky test detection.

• Public Dataset: We have made our entire dataset publicly available to foster further research and innovation in the flaky test detection community.

Empirical Results

Our empirical evaluation shows that, while only slightly dropping accuracy, CANNIER can decrease the time cost of rerunning-based flaky test detection techniques by an average of 88% across three different techniques. For instance, when applying CANNIER to the rerunning technique, we were able to reduce the time cost by 89% while maintaining a high Matthews correlation coefficient (MCC) of 0.92 for the correlation between the technique’s flakiness predictions and the actual flakiness labels. This result demonstrates that CANNIER is a practical and effective solution for developers struggling with the high cost of flaky test detection. Please read the paper for its many other empirical findings!

Future Work

The findings in this paper open up several avenues for future research. For instance, we plan to further investigate the features associated with test flakiness, potentially using causal inference techniques to gain a deeper understanding of the root causes of flakiness. We also intend to evaluate the performance of CANNIER on more specific categories of flaky tests. Since this paper’s experiments were done on open-source Python projects, we are also interested in applying CANNIER to both commercial software projects and programs and test suites written in other programming languages like Java.

Return to Blog Post Listing

Insights

I’m thankful that Hyrum took the time for this interview! Here are some key insights from it:

How does a project’s lifespan influence engineering decisions?

“As you write software, think about how long does this software have to last? Am I just going to throw this away tomorrow? Is it a bash script that’s going to exist on my command line, or is this something I need to have for the next 10 years? At Google, we actually think a lot about whether or not this is software that’s going to last for a long time.”

What role do constraints play in software engineering?

“Picking appropriate constraints as an organization scale is actually a really powerful way of enabling that kind of scalability. We kind of knew that going in, but it’s amazing to me how that came out in the book, whether it’s, again, large-scale changes or build systems or testing infrastructure or style guides. All those things benefit from having appropriate sets of constraints.”

What role do automated changes play in software engineering?

“Early on when we were doing a lot of these automated changes, people would push back, how do I know that this change is correct? Do I trust you? There’s a strong culture of stewardship within Google’s code base. Everyone can read all the source code, but specific individuals and teams are responsible for maintaining specific parts of the code base.”

Listen

In conclusion, if you’re interested in learning more about software engineering practices at Google — including testing, static analysis, large-scale code changes, and the importance of time, scale, and trade-offs — I highly recommend listening to Hyrum Wright on Software Engineering at Google from Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or you can listen to it with this handy podcast player.

Return to Blog Post Listing

Insights

I learned a lot from Zac during this interview. Here are some of his key insights:

What is property-based testing and how does it work with the Hypothesis tool?

“Property-based testing gives us a different attitude to what we check and a much richer set of tooling to help computers generate those inputs for us. … So when I think about writing software tests and the kind of unit test style, maybe they’re testing small units, maybe they’re testing whole programs. There are basically two problems that you have to solve in order to have a software test. The first is that you need to work out what kind of input data you run, and the second is that you need to check in some way that your code didn’t do the wrong thing.”

What challenges or limitations may developers face when using Hypothesis?

“The first thing I do wanna say before I get into the downsides is that I think almost every good test suite will involve some or maybe many example based tests. … The problem with basing your entire test suite on example based tests is just that it takes a long time and it gets really tedious to write them. … And so I think most of the downsides are a combination of requiring a lot of manual effort to define a rigorous test suite like this. And also that it’s really easy to miss the edge cases that you didn’t think of.”

How does Hypothesis work and how can developers effectively use it?

“What you do with Hypothesis is you write a test function which should work for any valid input and then you describe to Hypothesis what kind of inputs are valid. … And then Hypothesis can try to generate many valid inputs according to that specification you gave it looking for one which makes your test fail.”

Listen

In conclusion, if you want to explore property-based testing for Python programs and the Hypothesis tool, I strongly suggest that you listen to Zac Hatfield-Dodds on Property-Based Testing from Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or listen to it with this handy podcast player.

Return to Blog Post Listing

Insights

I learned a lot from Zach during this interview. Here are some of his responses to my questions:

How is the Warp terminal different from other terminals?

“The big sort of product innovations that we’re trying to bring to the terminal is to make it work not just for individuals, but for teams. And so one of the features I talked about earlier, the ability to take a block, get a link to it and share it to people on your team is the start of that.”

Why does the Warp terminal use the GPU?

“The primary reason that we use the GPU is performance. When you’re a developer using the terminal, there’s a lot of reasons why you might want to use the terminal, but one of the things I think that’s absolutely essential is that the terminal performs really, really well.”

Why did Zach and his team choose Rust to build the Warp terminal?

“Rust is a great language. It’s … I think you typically call it a systems language. It compiles down to binary … It has great community support and it has great cross-platform support, which is another really key thing for us.”

Listen

In conclusion, if you’re interested in learning more about the design and implementation of Warp or programming in Rust, I highly recommend that you listen to Zach Lloyd on Terminal Emulators from Software Engineering Radio! You can find it on your favorite podcast player, Apple Podcasts, Spotify, YouTube, or listen to it with this handy podcast player.

Return to Blog Post Listing

Ideas

The idea is that the SE book will help the students and I to learn the professional knowledge and skills that we need to develop as developers while the FB and DB books will equip us with the technical knowledge and skills necessary for us to both understand how software engineering tools are built and to build and apply our own software engineering tools.

In accordance with advice from well-known software engineers like Addy Osmani, who recently published a blog post entitled “Write about what you learn. It pushes you to understand topics better.”, the students and I will be writing a short précis about each of the articles in the Developer Development schedule. Today’s article, and the subject of the remainder of this post, is the preface of SE! This précis is written from the perspective of an instructor who wants to effectively teach undergraduates how to engineer software.

The preface of SE makes it clear that programming is not the same as software engineering — even though software engineering certainly involves programming! The authors of SE make the compelling point that software is different than programming due to the following concerns:

• Time and Change: while focused on solving problems with software, engineers must maintain software systems over long periods of time.

• Scale and Growth: all of the people within an organization who build and use software must adapt in both a daily and long-term fashion.

• Trade-offs and Costs: bearing in mind that software and people are always changing, and organization must make decisions informed by trade-offs.

The SE book’s preface explains that it will introduce how the culture, processes, and tools at Google help the organization to effectively create and deploy global-scale software. I’m hoping that the students and I can learn lessons from the engineers and managers at Google so that we can develop as developers! With that said, I see that, as an instructor, I face challenges on the time, change, and scale fronts. How can I ensure that students develop as software developers during a sixteen-week course? What is your advice for software engineering education?

Plans

I plan to introduce a number of my strategies through blog posts that I will write until the course concludes at the end of December 2023. With that said, here is an insight into one aspect of my current plans, which are documented further on the Developer Development web site. You will notice that I’ve integrated team-work into every conceivable aspect of the course, ensuring that students learn how to engineer software in an organization. Together, the students and I will run class sessions, write and publish articles, give presentations, assign software engineering tasks, and implement, deploy, and maintain software. We will work in pairs, small teams, and large-ish teams! So, is it possible to effectively teach software engineering, as defined in the SE book? I’m not yet sure — let’s see! Please check the course site and follow my blog posts for more details!

Return to Blog Post Listing