Google Lighthouse Part 1: Performance Testing

I have had a task to check out performance testing tools hanging out on my todo list for a bit. I did a quick pass through a number of different offerings, and hoped to give the Sauce Labs offering a look after attending SauceCon last year, but we aren’t enterprise customers, so that didn’t happen unfortunately. In the end, I kept getting drawn back to Lighthouse which is a free tool from Google. In their own words:

You give Lighthouse a URL to audit, it runs a series of audits against the page, and then it generates a report on how well the page did. From there, use the failing audits as indicators on how to improve the page.

There’s a number of ways you can run Lighthouse — there’s a web UI, you can run it in Chrome using an extension or you can run it from the command line. The command line option might come in handy in the future if we roll out an automated system for testing and tracking data over time. And there’s a cool way to share Lighthouse Reports using an online viewer and Github gists (you can see the report in it’s entirety that I’m screenshotting here).

One of the things that really got me interested in Lighthouse is that it does more than just performance testing:

It has audits for performance, accessibility, progressive web apps, and more.

In order to keep the length of this post under control however, I’ll focus on just the performance part in this blog post, and then write follow up posts about the different sections later.

Lighthouse summary

The top part of the report gives you a summary of how things went overall. Everything is a score out of 100 (yikes, we’ll get to the performance score in a second!) . You’ll notice the warning about possible Chrome Extension conflict here — I think that in this case it’s likely warning about the adblocker I run, so I did as suggested and re-ran Lighthouse in Incognito — the results were almost identical (the performance score was slightly lower, but that makes sense because it was making calls to GTM).

Performance Results

Performance report

The performance audit is broken down into four sections: Metrics, Opportunities, Diagnostics and Passed audits. Metrics gives us some numbers on how quickly things loaded and were displayed. There’s more to performance than just page load time — performance is really tied to visitor experience. As such, a lot of the numbers displayed here relate to how people see our pages. We didn’t do great in performance according to Lighthouse. For those interested in how the scores are calculated, here’s that information.

First Contentful Paint (FCP)

First contentful paint is the time at which the first text or image is painted (displayed) on the screen. In this case, from the time I clicked on the Invoices link to the time anything was displayed to me was 5.7 seconds. If that was the home page of, that would be really poor result, but it’s a pretty data-heavy page within the application so it’s probably okay. That said, Lighthouse is about to give us some suggestions on how to improve it!

First Meaningful Paint (FMP)

Unlike First Contentful Paint, First Meaningful Paint measures how long it takes before the main content of the page is displayed (you can see that both of our scores are the same). You might be scratching your head at this point wondering “what the heck is the difference?” An in-app page probably isn’t the best example for explaining the differences, so instead think about a news story on a popular news website. Those pages are frequently filled with advertising which, frustratingly, always seems to load first. On a page like that, we might might see an ad right away — within say 2 seconds (FCP) but it might take much longer for the news story to load, say 10 seconds (FMP).

In the case of the invoices page, all of the content is being loaded at once. We aren’t displaying (for example) summary information, followed by the bar chart, followed by a list of invoices. It’s all delivered together. We’ll talk about this a bit more under “Opportunities”.

First CPU Idle/Time to Interactive

This is basically a measure of “done” in our case. These metrics are measuring when the page’s main thread has quieted down enough that the page can be interacted with (a search could be performed, one of the invoices could be clicked on). In our case it’s around 7 seconds — which isn’t bad at all because this is a pretty heavy, in-app page.

Opportunities & Diagnostics

The majority of the opportunities identified revolve around removing unused JS and CSS from the page. Lighthouse estimates that we could save 1.5 seconds off our load time by cleaning up our CSS and deferring the loading of styles.

The biggest recommendation is in eliminating render-blocking resources. From the report, our fonts CSS file and our invoices JS file both weigh in at over 200 KB each. We’re requiring those to be fully loaded before we display the contents of the page. According to Lighthouse, there are 3.85 seconds of potential savings to be made in this area.

Under Diagnostics, the picture becomes even clearer. The top recommendation is to minimize main-thread work — it take 1.893 ms to evaluate the JS on the page and an additional 800 ms to process the style and layout of the page. By cutting down the complexity of the page layout, CSS and JS processed, we could see significant savings!


Just because we CAN do these things, don’t mean that I think we SHOULD do these things! I haven’t spent hours in the CSS and JS of the /invoices page or looking at the markup. I recognize that doing the development work on this stuff takes time, and nobody is complaining about the speed of this page anyhow — so why bother? This write-up wasn’t at all a critique of the Invoices page, but was only meant as a way to illustrate what types of information we can get from performance testing and what we can do about those results once we’ve got them!

Interviewing Testers

We’re hiring! We’re in the process of looking for a new team member in QA and there are some candidates that are nearing the interview stage. Panic! What are good questions to ask a candidate for a testing position?

Generally speaking, we’re looking for people who will fit in well with the team. So a big, important part of the interview should be asking yourself “is this somebody I can work with?” Can you have a conversation? Are they personable?

The second big question to ask yourself is “are they able to communicate ideas clearly?”. Some folks aren’t great talkers (*raises hand*) and are better communicating ideas through writing — that’s fine! We have a smoke test where we can see how well they communicate in writing. Everyone should be able to talk for at least a short while about a topic that interests them — so I think it’s really important to try to engage the interviewee and find out about their interests. Ask them about one of their hobbies or interests and listen to what they have to say.

Eventually, the discussion will have to turn to the technical side of things. So what are some good questions to ask somebody who is applying to test software? Luckily, this question came up recently in a Slack group that I’m part of and some folks linked to a couple of great posts on the subject.

The first, by Dan Ashby, pretty much sums up how I’ve approached things in the past. There’s a lot of topics in testing, so I’d have a mind map similar to what Dan included in his post. I wouldn’t try to check every box, but I would pick and choose from the various areas and ask questions about different concepts. The goal, again, is to try to have a conversation about a topic and hopefully try to surface some real-world experience in that area.

I think this approach is still valid, and so I’ll definitely be cherry picking a few topics in the upcoming round of interviews. The second post that I found really interesting is by Paul Holland and talks about his “3 killer interview questions”.

I have to admit that the first time I read through Paul’s post I thought that it was way too industry-specific. I thought that it required some knowledge of telecom networks (which is fine for Paul, because that’s where he was working). After reflecting a bit, I realize that’s not the case at all.

I’m not a big fan of his first question because it assumes a development process that’s very dissimilar to our own. We don’t produce builds and releases and have long stretches of time between them. Because that isn’t a world that we live in, I wouldn’t feel right asking questions about it.

Paul’s second question is one that I had written off as industry-specific, but in retrospect, it isn’t about that at all. He’s drawing a network diagram and asking the individual to troubleshoot a network problem. You don’t need in depth knowledge of TCP/IP in order to do that, you just need basic problem-solving skills! I may try out a version of this question in the upcoming interviews.

Finally, I really liked his third question. He talks about an online delivery service with an online order form. He first asks how the interviewee would test the logic behind that form, how many scenarios they’d need. He then asks for ideas/clarifications/improvements to the logic behind the form to improve business.

I really like this question because it not only dives into the analytical thought process, but also into creative thinking. Ultimately, our job isn’t to “build a form” or “build a report”, it’s to build a tool to help our customers run their businesses!

Documenting Exploratory Testing

Exploratory testing is important, whether it’s a new feature that we’re introducing or a bug fix that addresses a long standing issue. Doing a deep dive into our product to find any issues (new or longstanding) is a rewarding (and sometimes humbling) experience. It can be interesting to just take a fresh approach from time to time, and go through some of the happy paths our customers experience.

One of the key parts of exploratory testing is having a way to document your findings: all of the bugs, questions, notes and ideas that hit you while you’re clicking around the app. If you find a bug, but didn’t keep track of how you got to where you are, how useful is that to anyone? Isn’t it better to jot down an idea as you think of it?

I spent some time a short while ago searching for the *perfect tool* to record my exploratory testing sessions. It would be great if there was something out there that you could start and stop, would record what you’re doing in your browser, and allow you to annotate those activities with notes and observations.

I couldn’t find a tool that fit the bill perfectly. You can accomplish all of that manually by running a screen recording and keeping a set of notes. Just note the timecode on the recording when you’re making a note. That’s a bit cumbersome though.

Instead, I compromised somewhat and found a nice Chrome Extension that will allow me to jot down notes and bugs, and take screenshots as I go.

exploratory testing extension in action
Exploratory Testing Extension

When you’re finished testing, you can export a report of your testing session to CSV, JSON or view it in HTML (which you can then print to a PDF). It’s come in handy a couple of times in sharing my results back with the product team.

Mind Maps

One of my goals for the year was to slow down and think about/plan out the testing that I’m doing ahead of time. I’ve found that I can get too involved in the details of a task and miss seeing where the task might touch other areas. Asking questions like “will this change affect the API?”, “will this change impact the apps?” or even just “what are all of the areas that accept text input?” can come in handy.

I’m not a huge fan of generating a lot of documentation that will never get read again, but I am a visual thinker — so in order to flesh out ideas I need to write things down. In order to strike a balance here, I’ve started taking a few minutes before starting testing to sit down and put together a quick mind map of what might need testing.

My process for this is pretty simple. I’ll start clicking around that area in the app and brainstorming all of the parts that might be touched by this particular change:

mind map grouping of invoice form updates

I use an app called OmniGraffle. I’ll start dragging out squares for anything and everything. If there’s any possibility that it’s touched by this work, it gets a box. I’ll try to be as specific as possible at this point, but I often just use a shorthand to describe something (it’s for my own reference, so if I understand it, good enough!)

The next step is to group related bits together. It could be a grouping by function, or by location in the app — again, this is just for my own purposes to try to help give me some structure to my thinking. I’ll often notice other things while grouping and add them in (or break down bigger ideas).

Then it’s time to get going. If it’s an office-hours type of task, there may or may not be test cases. If there aren’t, and it’s a bit more complex feature, I may translate the mind map into a checklist so that I can keep track of where I am in testing things.