Speed up debugging of feature tests using screen recordings

Testing is an essential part of every software development process. Feature tests, in particular, verify that our system and its features aren't throwing tantrums behind the scenes.

However, the task of debugging these feature tests is often like trying to solve a mysterious puzzle with a clock ticking loudly in the background.

The complexity of feature tests, compounded by their notorious susceptibility to flakiness and with error messages that sometimes seem to be speaking alien languages, can make debugging feel less like intelligent problem-solving and more like combing a haystack for that sneaky needle.

The superhero of this article: video captures

Imagine being able to scrutinize those pesky tests in action, rewind or slow down the action to spot the villain.

At Botyglot, we use video captures of feature tests to fast-track our debugging process, leading to faster fixes and, let’s not forget, getting us off work in time for dinner.

Off-the-shelf solutions

SauceLabs and BrowserStack offer a video recording feature that captures test executions and facilitates real-time observation and debugging. However, as the saying goes, “all good things come with a price.” The high value brought by these solutions often comes with a hefty price tag.

We wanted to channel our inner Pareto and find a more cost-effective alternative. So, we ventured forth on our quest.

Testing screen-recorder gem

Turns out we weren't the first to address this issue in the ruby community, we simply rolled up our sleeves and decided to test the screen-recordergem in a project that was riddled with several flaky tests.

This approach paid off dramatically.

We were able to comprehend and diagnose the root causes of the flaky tests relatively quickly. As a result, we successfully managed to eliminate the flakiness, thereby enhancing the reliability and robustness of our tests. This strategy not only sped up our debugging process but also improved the overall quality of our project.

Adopting the approach for all our projects

From that moment, we incorporated this feature across all our projects, recording feature-tests on the CI, but discarding recordings of passed tests to manage costs.

Additionally, it boosts client trust as it assures a stronger and more reliable software system, made possible through the speedy resolution of any potential issues

Show me the code

Please read the readme of the gem for installation instructions, especially the installation of FFmpeg, the underlying recording library.

Once that is done, here is an example on how to use the gem

#  spec/support/screen_recorder.rb
require 'screen-recorder'
require 'fileutils'

RSpec.configuration.add_setting :screen_recorder_output_folder

RSpec.configure do |config|

  unless ENV['DISABLE_SCREEN_RECORDER'].present?

    config.screen_recorder_output_folder = File.join(Rails.root, 'tmp', 'screen-recorder')

    # Before each feature type test, we initialize and start the screen recorder.
    config.before(:each, type: :feature) do
      # Make sure the folder is screen recorder output folder exists
      FileUtils.mkdir_p(config.screen_recorder_output_folder) unless File.directory?(config.screen_recorder_output_folder)

      # Define the file path for the output video.
      timestamp = Time.now.utc.strftime("%Y%m%d-%H%M%S")
      @screen_recorder_output_file = File.join(config.screen_recorder_output_folder, "features-recording-#{timestamp}.mkv")

      # sempahore ci screen size is 1900x1200
      # browser window size is also 1600x1200
      advanced_config = {
        input: {
          framerate: 30,
          video_size: '1600x1200'
        },
        log: 'tmp/screen-recorder.log',
      }

      ScreenRecorder.logger.level = :DEBUG

      # Initialize the screen recorder with desired settings.
      @recorder = if ENV['DISPLAY'].present?
                    ScreenRecorder::Desktop.new(input: "#{ENV['DISPLAY']}", output: @screen_recorder_output_file, advanced: advanced_config)
                  else
                    ScreenRecorder::Desktop.new(output: @screen_recorder_output_file, advanced: advanced_config)
                  end

      @recorder.start
    end
  end

  # After each feature test, we stop the recorder and handle the output file as necessary.
  config.after(:each, type: :feature) do |example|
    unless ENV['DISABLE_SCREEN_RECORDER'].present?

      if @recorder.present?
        @recorder.stop
        if example.exception || example.metadata[:screen_record]
          # If the test failed, keep the recording and notify about its location.
          puts "Recording saved to #{@screen_recorder_output_file}"
        else
          @recorder.discard
        end
      end
    end
  end
end

This piece of code:

• will create a record for every feature test in the tmp/screen-recorder folder (except if the DISABLE_SCREEN_RECORDER env variable is missing)

• will keep the recording only if the test failed

• you can force keeping the record for a tests, even it the tests passes by adding screen_record: true to the test

After the tests suite has been executed, simply export the tmp/screen-recorder folder as artifacts and you will be able to review them.

Keep in mind the recordings have the format .mkv that can be read with a software like VLC.