We discuss here what tool to use for next release (4.3) to write test plans
testing first starts with writing usecases, so you know what to test and are not limited to a certain area of application that each developer is aware of. Use case contains main scenario and alternatives.
We have 3 areas:
GUI and API is esentially the same thing - they expose CRUD operations and few thing more. What really concerns me are the internal processes - like rating, invoice agregations, etc.. Maybe because I know little of that area.
Given the scope of our application, manual tests and testing with excell are not suitable - there are 5-10 of us working on core commiting 2-3 times per day. Imagine that with every commit you run the full test all over again manually - impossible.
The goal is to automate it.
I am not sure about GUI, since I have not seen how selenium works, but either they were missing result comparison, or very incomplete - I have stumbled on errors in GUI that it should have definatelly found - and I am talking about a simple button click in one of the main pages.
For API - we have covered all CRUD tests with assertions checking every field. SoapUI interface is like Windows 3.1, difficult to use, but we have it working. You no longer need to clean DB before each run - they have an autoincrementing index in entity code. So that part I would leave the way it is - after all it is little logic - it is like GUI - it takes input and shows result and you compare if it is what you expected or not.
Now, the tricky part are internal processes. You can not use GUI tools or API tools to test it. First you need to setup data (and it could be complex data), you can have same test with different values and result check could span multiple areas.
We have to use Junit/testNG/Arquilian to test it. In Bytel Project I had written complete test for file processing job in TestNG, and never had to come back to it. And job had asynchronous processing, etc...
So my sugested action plan is the following:
1. Write test cases for internal processes and jobs (Word)
2. Write test implementation for internal processes (Arquilian)
3. Review API tests to see if any API call is missing and implement them (SoapUI)
4. Write test cases for GUI (Word)
5. Implement test cases for GUI (dont know what tool - need to review Selenium)
Manaty Sebastian Michea:
about process testing : what is good with soapUI is that not only dev team can use it, vince also :) (and all business analyst too). those test suite are very often the source of demo or PoC. reciprocally test suite realized for Poc or Demo can be base of some non reg tests.
our consultants produce quite a lot of such test suites. it would be a loss of energy not to reuse them
Andrius what is your feedback about the google doc testplan ?It is a good start if you remove Status column. It has to be automated.
but if you look at contents - the actual save/update has minimal coverage in compare to other funcion - export to Excel, xml, reset form.
I would not polute test case with those items, and have them separatelly to test form reset, etc.. And maybe one sheet per menu. It is is all about details - you can write "update customer" or indicate exactly what values to change, expecting that it will trigger something else.
[08:49] Manaty Sebastian Michea: about process testing : what is good with soapUI is that not only dev team can use it, vince also :) (and all business analyst too)
when I talk about internal process, I mean things like price calculation, account or charge operation registration, runing job. SOAP UI is not for that. If you ever wrote more then one assert, you know it is not a tool for that. It is really, really, really crappy tool to use in terms of interface. I have 100 assertions in one Find request, and have no clue what they are about - need to open each to see what it is.
Or like job testing. - it quite complicated. How do you test it multiple times with different values? What if you dont have API to get to the value in DB that is not exposed as API - you write another API, or ignore it?
SOAP UI is just an execuse because you have clients data. But I am sure they dont cover even half what needs to be tested.
I have no time to spend to look for excuses, if you dont see the interest of SoapUI for consultants or non developpers doing QA pls ask them
If there is no API to get some value in DB you do have to write the missing API,
It is an important value proposition for opencell that everything can be performed by API.
Also you miss a point in this discussion, the best solution is probably not to do everything as arquilean tests or SoapUI tests but to find what is relevant to do with what.
Unit java test are ok if :
- more difficult to create and maintain with soap UI
- not necessary for non devs
2.) Testplan - Dev and QA will be the same set of resource? How will it work? If DevA develop feature A then DevB can test it and vice versa?
3.) What kind of test do we need to run? Should we also create load testing tests?
From what I can see we need:
A.) API - SOAPUI and Arquillian
B.) GUI - Selenium
C.) Internal Processes - TestNG / Arquillian, perhaps we can also plugin Mockito
D.) Load testing - JMeter or The Grinder?
But then before all of these we need to define first how we write our test plan and to what limit it will apply. For example, do we still need to create a test for a simple GUI update?
How do we track these tests?
we also need to write test on Meveo modules.. but we might postpone that to 4.4
Yes QA team = Dev team, i suggest that when a code reviewer valid the code it assign it create the test plan and assign it to a dev of his team. This mean that in term of organisation we would have :
Team leader with role : tech spec & ticket writing, code review, test plan writing, test review, documentation review.
Team member : dev & unit tests, functional, integration and non reg testing and automation, documentation writing
in terms of tool:
we could :
- Unit tests : use arquilean, the internal services tests Andrius was speaking about would fall into that
- Integration and functional tests : selenium and soapUI
- load test... i dont know
And to come back to the test plan : i think a google doc is fine
When I talk about internal processes, I talk about the inside code, something that should be unit tested. It might it be exposed as API or might never be, because it is much deeper/lower than end user would ever see.
SoapUI tests, that sales or customers have, correspond to integration or functional testing, and for that you should/could reuse it
As far as merging developer and tester. I am fine to do component level testing along with code review, but we still need someone that can do functional testing across the all system.
We should also consider time requirements - if I am working on a ticket that is complex, takes few days and requires some thought, I prefer to concentrate on it instead of getting distracted with code review and testing, which might require environment setup, etc..
Same happens when working on customer support when it is given more priority.
So code review and testing tasks might have a delay of few days.