SIP Signaling Regresson Test Tool
Introduction
This page describes a trace driven signaling test tool for sip signaling components. It was driven by the needs of SIpXbridge but can be adapted to regression test other components in the system. The tool that is described here is a developers test tool. The envsioned use is for a developer to run the tool on his component before commit.
Problem Definition
Regression tests for SIP signaling components are difficult to construct. Tools that rely on scripting expect the developer to construct a script to emulate signaling behaviour. Constructing a scenario that exactly emulates a signaling behavior can be difficult and error prone. Any test tool that relies upon individually crafted scripts to run the test falls victim to errors that may exist in those test scripts. Such test scripts may be useful when developing the system but not as a means of regression testing.
Our normal mode of development for the signaling components consists of developer driven testing for various call flows which is done against actual endpoints, gateways etc. This is followed by system testing by the QA team which runs the developed components in a pseudo operational setting by running various call handling scenarios.
For example in SipXbridge testing, we may test call transfer blind to local phone from ITSP, call transfer attended to local phone from ITSP, call pickup for inbound call or outbound call etc. Each of these scenarios generates a call flow.
As the system evolves and goes through interoperabilty testing, we would like to ensure that scenarios that have already worked. One way of doing this is to run tests manually against actual equipment. However it is exceedingly difficult to do this because it requires a lot of configuration. Further, due to the dynamic nature of the protocol, different message interleaving may result in the test case than in the scenario that was debugged and hence the developer is never sure that exactly the same scenario has been re-run.
Running the regression tool
In your BUILD directory under sipXbridge/regression, edit the tester.properties file to specify your IP address.
run
make regression
Principle Of Operation
Signaling services such as sipXbridge are message driven finite state automata. i.e. from an external perspective they are driven by SIP signaling messages that appear at a port and they generate messages as a result of such messages. The important thing with testing such components is to send requests to and expects requests and responses from the SUT in the same order as a working test case does. The regression is a stimulus response tester which tries to maintain the same message ordering as was observed in a trace. The diagram below shows the setup :
The regression test tool (RT) is meant to test a single component in isolation by emulating actual call flows that interact with that component. Emulation consists of presenting the SUT with a sequence of SIP requests which are inferred by reading a captured trace. RT reads a trace of a successful interoperability test that is presented as a suitably filtered "merged.xml" file (i.e. the output of sipx-trace with appropriate flags to isolate the call Id's of interest. It then consults a second configuration file (monitored-interfaces.xml) that indicates which IP addresses and ports in the captured merged.xml are of interest. A third configuration file (testmaps.xml) specifies an address mapping between a given trace message and an emulated trace message. As each message is presented to the SUT, addresses contained in the signaling such as the Via header address, Contact address, Route header address etc. are re-written to correspond to the corresponding address of the tester by consulting this mapping file. Finally, the tester address and base port are configured in tester config.xml
The next step in the process is to re-write the configuration for the SUT. Again, the trick here is to make all addresses of relevance be tester addresses so that the SUT can be run in a test environment.
After making the above transformations, the tester exercises the SUT by reading the trace file a message at a time and originating messages from emulated endpoints corresponding to the actual endpoints that were present in the trace gathering phase. For each request generated, the tester expects to see responses in a certain sequence and subsequent requests in a certain sequence. If such messages are not seen, the test fails. If such messages are seen in the expected order then the test passes.
It is important to note that the tester does not replay messages in timer order alone. The tester is event driven. It constructs a Directed Acyclic Event Graph (DAG) and a schedule in memory before running the test. Each DAG node represents a message to be sent (SIP request or response ) to be sent. A node of the DAG becomes ready when all events that are supposed to have happened before that node have been observed at the tester. The DAG represents a partial ordering of events and actions.
The tester can emulate proxy as well as Use Agent. Requests sent from the proxy server can have multiple Via headers. The tester preserves via header orderings and parameters while re-writing the parameters of this header.