Speed up debugging of feature tests using screen recordings
Testing is an essential part of every software development process. Feature tests, in particular, verify that our system and its features aren't throwing tantrums behind the scenes.
However, the task of debugging these feature tests is often like trying to solve a mysterious puzzle with a clock ticking loudly in the background.
The complexity of feature tests, compounded by their notorious susceptibility to flakiness and with error messages that sometimes seem to be speaking alien languages, can make debugging feel less like intelligent problem-solving and more like combing a haystack for that sneaky needle.
The superhero of this article: video captures
Imagine being able to scrutinize those pesky tests in action, rewind or slow down the action to spot the villain.
At Botyglot, we use video captures of feature tests to fast-track our debugging process, leading to faster fixes and, let’s not forget, getting us off work in time for dinner.
Off-the-shelf solutions
SauceLabs and BrowserStack offer a video recording feature that captures test executions and facilitates real-time observation and debugging. However, as the saying goes, “all good things come with a price.” The high value brought by these solutions often comes with a hefty price tag.
We wanted to channel our inner Pareto and find a more cost-effective alternative. So, we ventured forth on our quest.
Testing screen-recorder
gem
Turns out we weren't the first to address this issue in the ruby community, we simply rolled up our sleeves and decided to test the screen-recorder
gem in a project that was riddled with several flaky tests.
This approach paid off dramatically.
We were able to comprehend and diagnose the root causes of the flaky tests relatively quickly. As a result, we successfully managed to eliminate the flakiness, thereby enhancing the reliability and robustness of our tests. This strategy not only sped up our debugging process but also improved the overall quality of our project.
Adopting the approach for all our projects
From that moment, we incorporated this feature across all our projects, recording feature-tests on the CI, but discarding recordings of passed tests to manage costs.
Additionally, it boosts client trust as it assures a stronger and more reliable software system, made possible through the speedy resolution of any potential issues
Show me the code
Please read the readme of the gem for installation instructions, especially the installation of FFmpeg, the underlying recording library.
Once that is done, here is an example on how to use the gem
# spec/support/screen_recorder.rb
require 'screen-recorder'
require 'fileutils'
RSpec.configuration.add_setting :screen_recorder_output_folder
RSpec.configure do |config|
unless ENV['DISABLE_SCREEN_RECORDER'].present?
config.screen_recorder_output_folder = File.join(Rails.root, 'tmp', 'screen-recorder')
# Before each feature type test, we initialize and start the screen recorder.
config.before(:each, type: :feature) do
# Make sure the folder is screen recorder output folder exists
FileUtils.mkdir_p(config.screen_recorder_output_folder) unless File.directory?(config.screen_recorder_output_folder)
# Define the file path for the output video.
timestamp = Time.now.utc.strftime("%Y%m%d-%H%M%S")
@screen_recorder_output_file = File.join(config.screen_recorder_output_folder, "features-recording-#{timestamp}.mkv")
# sempahore ci screen size is 1900x1200
# browser window size is also 1600x1200
advanced_config = {
input: {
framerate: 30,
video_size: '1600x1200'
},
log: 'tmp/screen-recorder.log',
}
ScreenRecorder.logger.level = :DEBUG
# Initialize the screen recorder with desired settings.
@recorder = if ENV['DISPLAY'].present?
ScreenRecorder::Desktop.new(input: "#{ENV['DISPLAY']}", output: @screen_recorder_output_file, advanced: advanced_config)
else
ScreenRecorder::Desktop.new(output: @screen_recorder_output_file, advanced: advanced_config)
end
@recorder.start
end
end
# After each feature test, we stop the recorder and handle the output file as necessary.
config.after(:each, type: :feature) do |example|
unless ENV['DISABLE_SCREEN_RECORDER'].present?
if @recorder.present?
@recorder.stop
if example.exception || example.metadata[:screen_record]
# If the test failed, keep the recording and notify about its location.
puts "Recording saved to #{@screen_recorder_output_file}"
else
@recorder.discard
end
end
end
end
end
This piece of code:
will create a record for every feature test in the
tmp/screen-recorder
folder (except if theDISABLE_SCREEN_RECORDER
env variable is missing)will keep the recording only if the test failed
you can force keeping the record for a tests, even it the tests passes by adding
screen_record: true
to the test
After the tests suite has been executed, simply export the tmp/screen-recorder
folder as artifacts and you will be able to review them.
Keep in mind the recordings have the format .mkv
that can be read with a software like VLC.