Zephyr Essential - Test Management for Jira

Zephyr have a few design flaws which at first I thought they were defects - and I actually logged issues for the Global Admin to contact Zephyr's support team. But I was told they were by design! 1) If executed test steps are all Passed or all Failed, it automatically shows 'Passed' or 'Failed' respectively in the overall execution status. If executed test steps are not all Passed or all Failed, the overall execution status is shown as 'Unexecuted'. I wish to see the overall execution status displayed the 'Failed' status if any step in the test case fails. 2) A test case is always in the 'Open' status in the workflow. I have no idea why there is a workflow there - even though there is no option anywhere to go to the next step in the workflow?!? 3) In the Test Details section, both 'Test Step' and 'Expected Result' field are quite narrow. I wish to see the width of these fields user-customized so that we can expand to write more text in the field without going to the next line. (Maximum is only 46 characters per line.) 4) No version control upon modifying a test case. If the requirement changes from a release to the next, and you want to modify the test case, the old version is overwritten by the new version with the new text. There is no way to go back and see what was changed in the test case - and what test step had that change. History tab provides the general information - having the time stamp when the change was made and who changed it. And that was it. I wish that Zephyr history tab could capture the steps where modified text were made in a test case and have some sort of crossed text for the old test case version. 5) All test cases display in a flat list. If you want a folder structure for your test plan, you need to use 'Structure' grid to organize your test cases (We have around 3000 test cases.) Overall, I would say that Zephyr is ok for our QA team to use but it lacks of the usability concept. It serves the basic need of creating a basic test case and test execution cycles.

• 

  SmartBear

  We welcome you to check out our latest version - this and the previous version have a whole host of new features added in the areas of productivity, efficiency, and usability. We continue to add more to Zephyr and we look forward to hearing from our customers. The Team at Zephyr

Very promising tool, great to use from within Jira, our customer has decided to buy it for all-enterprise use. The lack of two major features is however quite painful: it would be really appreciated to have the possibility for bulk execute test cases and introduce hierarchy between test cases. We plan to use the tool for huge projects in the banking industry with hundreds or even thousands of test cases for each project. One blocking error may cause hundreds of test cases to go to "Blocked" state in a cycle, and it is really painful having to click through all of them. Otherwise the concept is great, looking forward to receiving new features in the future!

• 

  SmartBear

  Brigitta - your review is timely. Today at the Atlassian Summit we announced Zephyr for JIRA 2.0 (see http://blogs.yourzephyr.com/?p=2859) and it now supports bulk execution of tests, besides a host of other features.

We use the this addon one year and are disaapointed: - the tool missed a lot of standard features: - test repository. The possibility reuse the tests accross the jira projects - you can't add the testexecution to the sprint, only test cycle - you can't moved the test to the other projects, only if you have more rights (bulk operation) - comment field in the testexecution is not adjustable - testexecution is not a standard Jira issue, because of that you cant use that in other addon, for example Bigpicture for the Tetsplaning - the update policy is bad. the last feature updae was a year ago v 4.0. the version 5 is only compatibility to Jira 8 - test suppport is bad, beacuse they cannot do anything. The say only we are appriciate for your feedback and we give aour request to the project management and than it happens nothing. We can't recomended this tool at all and we want move to other better tools.

• 

  SmartBear

  Hello Sergei,

  We appreciate you providing feedback. At SmartBear, we aim to provide clarity in our roadmap for Zephyr for Jira since we plan to develop some exciting features based on customer input. We won't be able to incorporate everyone's feedback but our Product Team still evaluates all of it to come up with the best road map possible.

  Our roadmap can also be viewed here - https://trello.com/b/awC51iQh/zephyr-for-jira-server along with popular requests that can be voted on.

  We are happy to discuss this further with our Product Team. Our Atlassian Ecosystem Manager, Steven (steven.colon@smartbear.com), will be in contact with you to schedule this.

This is a nice starting point for centralised testcase management in JIRA especially if you have all the projects managed through JIRA. I love the test execution dashboard which provides an excellent overview of the testing progress. A few things that are still lacking are - Can't clone tests across projects. Can only clone tests from the same project. - Colour highlight to distinguish different test scenarios would be a great feature. The test steps doesn't support jira special characters such bold, italics,headers etc. It would be a nice to have for long test cases - Provision to add steps in the middle rather than type in the end and move. This is sometimes annoying as I have to go all way down to add steps and then move it to the middle. Also moving is not very scroll friendly. - It would be neat if a test case is marked as pass, all the test steps under it should automatically be marked as pass. - APIs to directly update test results on jira rather than manual execution. This is a must as it saves a lot of time marking tests manually as pass/fail. Very much waiting for the next release with one or more of this features covered. APIs would be great and very much waiting for it.

• 

  SmartBear

  Thank you for your kind words and feedback. Stay tuned - lots of new announcements coming shortly!

  The Team at Zephyr

  [UPDATE] Please check out the latest version of Zephyr for JIRA 2.x and new Add-on ZAPI for some of these features.

Ok at what it does but not for the asking price or compared to some other QA products on the market place that are cheaper to boot. In addition to T.L.'s review would like to add the severe absence of user permission control is baffling in this product. Any user with access to your JIRA project can go in and create test cycles which is a huge turn off when trying to tweak and secure a JIRA instance for a large Enterprise. Anyone who can see your Zephyr test case can also modify the test steps portion no matter what security permissions are placed on an issue itself. How Zephyr thought this was a good idea is beond me and leads one to believe that they're really just pushing their other product which is a fully stand alone suite that connects back to JIRA (a solution our devlopment team is trying to avoid). Another downside is the fact none of the statics used within it can be used by JIRA's gadgets, however this I believe is common for most plug-ins and is more a matter of getting customer demand to add additional metrics as needed or manipulating the use of workflows to get what is needed for your project. A lot of this sounds harsh and could be due to unforseen limitations of the JIRA API, however similar products like Tempo for time tracking (a product that has a similar API layout like Zephyr) which Atlassian also backs and are in this price bracket do go above and beyond what you'd expect a high price add-on to be. That includes adding full on permissions contol and hidding whole sections from people who should not see them to begin with. Again good for basic test cycle execution but may want to take another look at other tools that offer more customizations.

• 

  SmartBear

  Hi Pam, Thank you for your comments. The tack we have taken when it comes to permissions et al is to allow the add-on to be used by as many users as possible without having to involve the JIRA Administrator every time someone needs to create//run/view a test. Zephyr's functionality itself is quite unobtrusive and not in-your-face (we've taken a lot of effort to ensure that our UI melts seamlessly into JIRA's). We also provide ample testing metrics and continue to add more to our product with every release. Regards, The Team at Zephyr