What is testing?
- Testing is about knowing that your code works the way you want it to work.
- It’s about confirming the integrity of your code.
- It’s about ensuring that your code will work no matter what you throw at it. (Caveat: Something will always come up. Murphy’s Law.)
Many people will say that you don’t have to test. Good code can be written without instantiating testing suites. However, as an app, or any software grows, testing can confirm that newer pieces of code and older pieces of code are working well together -- and can catch and identify bugs easier and quicker. To this end, there are three basic testing paradigms:
- Test-Driven Development (“TDD”)
- Behavior-Driven Development (“BDD”)
- Design-Driven Development (“DDD”).
The Three Testing Paradigms
TDD - Test Driven Development
Test-Driven Development is a more traditional development process, where code is developed based on pass-fail testing process. Once it passes all tests then it is implemented into the source code for that software.
The advantage of TDD is that the testing suite is built as the code develops. The less-advantageous aspect of TDD is that code is written twice. This can be daunting to many and should be. It should, also, however, make sure that your code is written as succinctly and efficiently as possible.
BDD - Behavior-Driven Development
Behavior-Driven Development focuses on the user’s behavior as the template on which the code is assessed as “good” or “correct”. BDD is used in “agile” and “scrum” environments, where project and business managers have a need for more direct insight into the development process. In these environments, non-developers can see when the “behaviors” are fulfilled without looking at the code directly. On the other side, developers have a clear idea of what managers or users expect.
DDD - Design Driven Development
Design-Driven Development focuses on the design of the project. The developer implements code that fulfills the design specifications given. This is really used all the time -- as a software idea is generally not ---
“oooh! I love this code, let me design something I can use it for.”
but more like ---
“I have this cool idea! How would I code it?”
Summary: All these paradigms of testing can be used at any one time, or phases in a project. People do seem to defend their favorites. I am still agnostic.
Testing Code vs. Testing User Interface 
When deciding which testing paradigm to follow, there will have to be a balance between wanting to test specific objects and methods in your code or wanting to test the user’s interaction with the app, or the functionality of a design.
XCTest
With XCTest, the testing suite built into XCode (since 5.0) is great for test-driven development.
Pros: XCTest can access a method in source code.
Con: it cannot access the outcome, if it exists in another object in the code.
Neutral: One has to create mock objects and methods, such as CoreData, CLLocation manager, or simple “Company” or “User” objects, for the test to interact with.
There is an XCode UIAutomation suite in “Instruments”, which gives a lot of information about the code. I still have hope to get a greater handle on it. The interface is well-designed, and can also give you a ton of information about your code, memory management, timing and threads. The functionality for UIAutomation seems to work best, if the tests were created while building your code. Not after.
Basically: XCTest and UIAutomation seem to work best as part of a TDD suite.
frank-cucumber
frank-cucumber uses the language that BDD already understands. The test directions are written as a story about what the user is experiencing without programming language. It then seems almost a natural that the tests are written in Ruby which is known for it’s readability. frank-cucumber cannot test specific objects or methods in your code. It will only test the user’s experience.
Important: If the user is unhappy, the software has failed.
Why frank-cucumber?
The project I'm working on has chose frank-cucumber because there is a need to create a testing suite as this app grows. However, the vast majority of the code has already been written. So, in order to not copy the work that has already been done, it was decided to create a testing suite that interacts with the app. frank-cucumber fulfills that requirement easily.
Pros
Besides our project's development stage, the advantages of frank-cucumber are:
(1) the ease and depth of the user-interface interaction and
(2) the ease with which non-developers can understand the testing results.
Cons
- It is completely separate from XCode and has no Objective-C language. In fact, the guy that created and is managing frank-cucumber doesn’t know XCode or any Objective-C. Because of this, it can only see aspects of the app that the user actively interacts with.
- System call-backs which, for example, ask for location information are not accessible to frank-cucumber.
- It has no access to app objects, or even to create its own objects. It has been built to only interact with user-driven events in an app.
- The testing code has to be written in another language - cucumber - which is based on ruby. This is an advantage or disadvantage depending on how much ruby you already know and how good you are at picking up other languages. I think this is fun!
Conclusion
For an iOS testing suite, it has little in common with iOS language and development process, but it does provide a powerful UI testing suite. “frank inspect” is pretty cool - as you will soon see.
Frank resources
There are few resources for frank-cucumber, but they provide a tight circle of resources to go to for help. No need to scour the internet.
This is the official website for frank-cucumber. There isn’t much information, but the “getting started” information is very useful.
This is the guy who maintains frank. His website is very useful. The “getting started” section of the official website takes you to his blog. Nice!
Just as in apple, iOS or Java, there is documentation on the available methods, properties and objects that frank-cucumber uses.
This discussion group is monitored by Pete Hodgson. If you have a question, search here. The answer could already be amongst previously asked questions. Otherwise, ask your own. He responds in about 24 hours.
Frank Setup
Pre-note on “gems”
Ruby environment managers are pieces of software that allow one to install, manage and work with various ruby environments, including gems, which are used to install and manage all aspects of “frank-cucumber”, including the “console” which is based on “pry,” which has its own gem. It is important to use a ruby environment manager.
“rbenv”
We initially started with rbenv, which is considered “newer” and “better”, but had problems getting the “pry gem” to install and work with frank-cucumber. So we switched to “rvm”.
“rvm”
“rvm” is actually recommended by Pete Hodgson. It is unclear why one ruby environment manager would work better than another, especially when one is considered “old”. Here is the link for instructions on how to install rvm (make sure to first uninstall rbenv or other ruby environment manager): https://rvm.io/
Let’s Begin
Terminal - Bash
frank-cucumber is run and managed through Terminal - Mac OS X Bash. It is important to know how to navigate in bash and vim. How to go to a directory, list the files in a directory, remove and copy files to other directories and create files. Here is a link to a nice summary of commands: http://www.tldp.org/LDP/intro-linux/html/sect_03_05.html
gem install frank-cucumber
- Go to project directory: In bash, cd into the folder/directory that contains your XCode project that you want to test (the .xcodeproj file).
- type gem install frank-cucumber: if you have installed “rvm”, then type as given. If using another ruby environment manager - type “sudo gem install frank-cucumber”
- type frank setup - You will see a bunch of files being created.
- type frank build - More files being created.
- type frank launch - This launches the “Frankified” version of your app in the simulator. You can now interact with your app and see what it does.
- type frank inspect - This opens up a link to the frank server in Safari. It looks like this:
On the right, “View Hierarchy” is open and that some of the objects have a “:[text]” after them, such as “:Login” and “:Sign Up”. These are the accessibility labels that frank-cucumber uses to create items to test. Even if they aren’t explicitly created in your XCode project, it gathers information from labels and other properties which can be found in the Interface Builder.
“Accessible Elements” - if you click here, it gives you a list of the “views” or “subviews” that frank-cucumber can interact with in the written tests.
Note: you can interact with everything in the app in frank launch. When writing tests, only “accessible elements” can be interacted with.
On the left, View Locator is open. If you click on “live”, the image will change as the app changes, and the View Hierarchy and Accessible Elements will change accordingly. This allows you to find elements to test. Useful!
Below, an item in View Hierarchy is clicked and View Properties is also clicked. This time it is the “email text field”.
7. type frank console -
You enter the frank server to your app and you can navigate through your app by typing in cucumber commands in Bash. Useful!
It is great way to test if your written test code. If there are problems with your test, it is useful to know if it is your code or the way your code is interacting with your app.
For example - type:
touch “view:’UIButton’ marked:’Login’” to get from a home screen to a login screen.
Note 1: check to see that these are the accessibility labels you should be using!
Note 2: spacing and punctuation are important here!
To exit - type “exit” or control-d
Write a cucumber test
Now it is time to write a test. But first take a look at what frank has done in your project’s folder.
build
In here there are a few folders regarding build, which I haven’t needed to touch. There is one folder, which seems interesting:
~/appdirectory/build/YourAppName.build/Debug-iphonesimulator/YourAppName.build/Objects-normal/i386/ contains items regarding the build of every single object in your project.
Frank
“Frank” folder has three folders: “frankified build”, “features”, “plugins” -- “plugins” is empty for me, but it should be noted.
frankified build
There are two executable files in there.
YourApp Frankified (THIS is what is launched when you type frank launch.)features
This folder contains three important items: .feature files, step_definitions folder, support folder
my_first.feature (This comes with your build and allows your test to launch the app, and test for various device orientations.)
These are the your tests. Each “Feature” contains at least one “Scenario” with various steps.
step_definitions
This folder contains .rb (ruby) files in which you code the steps for the .feature files in cucumber.
launch_steps.rb (this is the .rb file for “my_first.feature) support
This contains the env.rb file which has general information on your frank environment. The instructions I received said I had to add a APP_BUNLE_PATH to this file, but it was already there when I opened it. If you have issues with frank finding your frankified app, go here.
env.rb
Now, let’s write a test!
Using “sudo vim” create new .feature file. DO name the file something that reflects what you want to test. My first was called: inputlogin.feature -- Pete Hodgson’s first is: navigation.feature
sudo vim Frank/features/newtest.feature
Quick sudo vim tutorial
To start typing and editing - go into “Insert” mode - type: i
To exit edit Insert Mode and go into Command Line mode - [esc] key
Command Line tricks
dd - delete whole line
shift-o - insert line above and enter “Insert” mode.
:w - save or “write”
:q - quit
:wq - save and quit
(p.s. “sudo” will mean that the bash screen will ask you for your computer’s admin password)
Basic format of a test description
A test has 5 basic components:
- Feature
- Scenario
- Given
- Then
- When
Feature: Navigating between screens
Scenario: Moving from the ‘Home’ screen to the ‘Events’ screen
Given I launch the app
Then I should be on the Home screen
When I navigate to “Events”
Then I should be on the Events screen
(purple is my emphasis)
Using sudo vim type a Feature and a Scenario with steps that you want to cover for a basic navigation scenario.
Run the feature
Quit out of vim.
Type - cucumber Frank/features/newtest.feature
“cucumber” will not be able to run the test, as there is no code behind these steps. But… ta da!
It prints out your test in yellow text with some extra information. THIS is the template for your step definitions. It is as simple as that. (purple text is my emphasis)
Given I launch the app # Frank/features/step_definitions/launch_steps.rb:5
You can implement step dfinitions for undefined steps with these snippets:
Then /^I should be on the Home screen$/ do
pending # express the regexp above with the code you wish you had
end
When /^I navigate to “(.*?)”$/ do |arg1|
pending # express the regexp above with the code you wish you had
end
Then /^I should be on the Events screen$/ do
pending # express the regexp above with the code you wish you had
end
1 scenario (1 failed)
4 steps (3 undefined)
The Given I launch step is already defined in your features folder through frank build. It should not come up as something that needs to be defined.
Write step definitions
Copy and paste the yellow text into…
sudo vim Frank/features/step_definitions/newtest_steps.rb
The format of the filename is important… “_steps.rb”
Note: Use the frank documentation to create the steps you want to simulate.
Then /^I should be on the Home screen$/ do
check_element_exists “view:’UIImageView’ marked:’Home”
end
When /^I navigate to “(.*?)”$/ do |arg1|
touch “view:’UIButton’ marked:’#{arg1}’”
end
Then /^I should be on the Events screen$/ do
check_element_exist “view:’UIButton’ marked:’Events’”
end
Much of this is an extrapolation from Pete Hodgson’s website, where his sample test seems to drill down the view hierarchy. I have found the “drilling” down format to cause my tests to fail, but I will include them as a footnote, for comparison. Depending on how the view’s are constructed, there should be a project for which they are appropriate
Run test
Type - cucumber Frank/features/newtest.feature
...after you have written each step - checking that each step works before you move onto the next one. Your code will be written in various colors to mark their status.
Given I launch the app # Frank/features/step_definitions/launch_steps.rb:5
Then I should be on the Home screen # Frank/features/step_definitions/newtest_steps.rb:5
When I navigate to “Events” # Frank/features/step_definitions/newtest_steps.rb:9
Then I should be on the Events screen # Frank/features/step_definitions/newtest_steps.rb:13
Failing Scenarios:
cucumber Frank/features/inputlogin.feature:3 # Scenario: When I navigate to “Events”
1 scenario (1 failed)
4 steps (1 skipped)
The above is an example of steps passing, failing and being skipped. This is when you go back into your code via sudo vim - refactor and see where the code is insufficient.
The color key is easy:
Red = fail
Gold = no step definition (copy and paste text into the .rb file and refactor)
Green = pass
Blue = skipped (due to previous steps’ fail)
The goal: In the cucumber community is all green - cuke! - like a cucumber, apparently.
Given I launch the app # Frank/features/step_definitions/launch_steps.rb:5
Then I should be on the Home screen # Frank/features/step_definitions/newtest_steps.rb:5
When I navigate to “Events” # Frank/features/step_definitions/newtest_steps.rb:9
Then I should be on the Events screen # Frank/features/step_definitions/newtest_steps.rb:13
1 scenario (1 passed)
4 steps (4 passed)
how to fill text field in StepDefinitions
ReplyDelete