Yeti: YUI’s Easy Testing Interface

Reid Burke

Hello, world!

Hello, world!

You’re either missing out on bugs or struggling with complicated tools.

Frontend Test Runners

The incumbents:

  • Selenium
  • TestSwarm
  • JSTestDriver

Focus: After The Commit

  • Selenium
  • TestSwarm

Intended to be used with infrastructure dedicated to testing.

Yeti is hyper-focused on before the commit.

Yeti is personal.

Frontend Test Runners

  • Selenium
  • TestSwarm
  • JSTestDriver

JSTestDriver works well before the commit.

But JSTestDriver didn’t fit our needs.

JSTestDriver requires JavaScript test files.

HTML is supported with their HtmlDoc feature.

We wanted to run our existing tests.

We wanted to run HTML documents with YUI Test tests.

Works with HTML?

  • Selenium
  • TestSwarm
  • JSTestDriver

It’s a TODO item for JSTestDriver.

JSTestDriver requires tests to be executed with their test framework.

Why not JSTestDriver?

No support for HTML documents. HtmlDoc is awkward.

We like YUI Test.

We wanted to use our existing tests.

Frontend Test Runners

  • Selenium
  • TestSwarm
  • JSTestDriver

Why Yeti?

Yeti works with unmodified YUI Test-based tests.

Which kinds of tests?

  • Unit tests?
  • Functional tests?

Whatever you want: It’s up to you.

That’s implementation details, not up to the test runner.

Yeti is fast.

Need for Speed

  • Feedback as quick as possible.
  • Advances to the next test immediately.
  • Runs tests in parallel with multiple browsers at once.

You’re testing closer to the change.

You run Yeti from the command line.

$ yeti src/async-queue/tests/async-queue.html [etc] Waiting for results. When you're done, hit Ctrl-C to exit. Y.AsyncQueue on Safari (5.0.2) / Mac OS 18 passed, 0 failed Cache Test Suite on Safari (5.0.2) / Mac OS 24 passed, 4 failed in Event Tests testAdd: Unexpected error: resume() called without wait(). testFlush: Unexpected error: resume() called without wait(). testRequest: Unexpected error: resume() called without wait(). testRetrieveSuccess: Unexpected error: resume() called without wait(). Attribute Unit Tests on Safari (5.0.2) / Mac OS 106 passed, 0 failed Y.AsyncQueue on Internet Explorer (9.0) / Windows 18 passed, 0 failed Attribute Unit Tests on Internet Explorer (9.0) / Windows 106 passed, 0 failed Y.AutoComplete on Safari (5.0.2) / Mac OS 50 passed, 0 failed

Yeti is fault tolerant.

Yeti recovers from syntax errors and 404s.

Yeti reports uncaught exceptions.

$ yeti test.html Waiting for results. When you're done, hit Ctrl-C to exit. test.html on Internet Explorer (9.0) / Windows 0 passed, 1 failed in window.onerror handler (yeti virtual test) window.onerror should not fire: Syntax error

Yeti reports tests not in Standards Mode.

$ yeti test.html Waiting for results. When you're done, hit Ctrl-C to exit. test.html on Internet Explorer (9.0) / Windows 0 passed, 1 failed in window.onerror handler (yeti virtual test) window.onerror should not fire: Not in Standards Mode!

Fault Tolerance

  • Syntax errors
  • Nonexistent tests
  • Uncaught exceptions
  • Files without tests
  • Not in Standards Mode

Yeti keeps you away from a web browser.

Yeti keeps you closer to your editor.

Yeti keeps you writing code.

When testing is simple, you spend more time on more important things.

  • Writing more tests
  • Writing more code

You spend less time on boring things.

  • Avoiding tests: what’s the point?
  • Launching tests yourself

Food for Thought

Graded Browser Support is a QA philosophy. It exists because you can only test so much.

It trades off testing everything for testing a majority.

What if you could test everything?

Your app will be more awesome if testing is easy.

Your app will be more awesome if you use Yeti.

Yeti is designed for the mobile future.

Yeti: Recap

  • Test as close to the change as possible
  • Your tests are grown-up HTML documents
  • Works with unmodified YUI Test-based tests
  • Fast test execution
  • Fault tolerant
  • Designed for mobile

The case for Yeti.

Testing the hard way adds up.

What tests can be made easy?

What tests can be automated?

Automate?

Maybe.

User interface → best tested by a human.

Most everything else can be automated.

Automation

YUI Test is a great web app testing framework.

  • Mocks
  • Async Tests
  • foo should do X
  • Works with any other framework

If a test can be automated, you should be using a testing framework.

If you’re using a test framework, you should automate test launching too.

Automation vs. Launching

Yeti is a test launcher.

It launches tests.

For automated tests, it automatically reports results to you.

Reloading just to view YUI Test results is a drag.

And, it doesn’t scale.

You need test automation.

You need a way to launch tests.

How does Yeti fit in?

  • Selenium
  • JSTestDriver
  • TestSwarm
  • Yeti

Using Selenium is complicated.

Selenium has its place in CI.

Selenium doesn’t work for mobile devices.

TestSwarm requires dedicated LAMP infrastructure.

JSTestDriver excels at performance at the expense of too many features.

  • Test closer to the change.
  • Test without infrastructure.
  • Test quickly.
  • Test what you already have.

Yeti is a great fit.

Let me show you.

Try running tests in your web browser or mobile device.

http://reid.hacked.jp/

A Listing of JavaScript Web App Testing Tools Written In JavaScript

  • Yeti

Yeti is written in JavaScript.

Yeti runs on Node.js.

Yeti Testing

Yeti itself benefits from asynchronous behavior driven tests.

Uses the Vows framework. http://vowsjs.org/

Practice what you preach.

One more thing.

Yeti is excellent at automated tests.

What about manual tests?

Automation?

Some tests are difficult to automate.

You’re still opening tests in dozens of browsers.

The test is your web site.

But you should be able to launch them automatically.

Stickshift

Experimental branch on my fork at http://github.com/reid/yeti.

Designed to launch manual tests. Compliments automated tests.

Stickshift builds on the fact web tests are web sites.

Supporting HTML is key to that ability.

Yeti makes you a better engineer today.

Nevertheless, we’re not done. Expect more.

Yeti 0.1.2

Released today. Better fault tolerance and bugfixes.

Getting Yeti

Exploring Yeti

We want you to make Yeti even better.

You're either missing out on bugs or struggling with complicated tools.

You’re either missing out on bugs or haven’t used Yeti.

Q&A

Yeti by @reid — YUIConf 2010