Category Archives: Uncategorized

randomtest-stdin: automated log scanner

Randomtest.net discovers crashes and special events (raised by SIGUSR1 in C/C++) raised from the software so far. But some kernel – level events (a warning from kernel module as an example) were not visible so far.

Randomtest.net has been extended recently with the ability to discover invalid state in logs (since 6a03f1f1). By invalid state I mean: kernel ops reported in dmesg, critical errors raised by kernel modules and so on.

randomtest-stdin reads stdin until empty newline is present then reports such text as it’s done already with textual stacktraces.

randomtest-stdin sample usage:

export RANDOMTEST_URL=”…”
export RANDOMTEST_VERSION=”…”
awk ‘/pattern/ { print; print “”; }’ /var/log/syslog | randomtest-stdin

As you can see the invalid pattern detection is externalised to custom awk script – randomtest-stdin doesn’t know what is valid or wrong. You are free to choose it by regular expression.

RANDOMTEST_VERSION: a method to save version infotrmation with stacktrace

When you have your Distributed Continuous Integration tooling enabled you get reports from various sources. Some of them might be not up to most recent software version. It’s important to distinguish crash from old version of a software from a regression in recent version.

Thus randomtest has been extended with version reporting ability. Each stacktrace has associated version list that emitted that stacktrace. So you can easily distinguish between old version and a regression in new one.

Implementation is easy (C/C++ application there): you set:

export RANDOMTEST_VERSION=`cat /path/to/version/file`

And every event would be reported with version information and aggregated reports would show that information.

HTTP random client driver – existing solutions review

I’m searching a solution (or just a library) that will allow to deliver random driver for HTTP-based applications. Ideally the tool will cover randomly state space of UI with random input to every HTML control found on page.

Firstly, lets review options that do not support JavaScript, only forms + HTTP functionality:

No JavaScript support

WebInject

webinjectWebInject is a perl-based tool using XML files for driving HTTP-based tests. Sample test specification:

<case
    id="1"
    description1="short description"
    description2="long description"
    method="post"
    url="http://myserver/test/login.jsp"
    postbody="username=corey&password=welcome"
    verifypositive="verify this string exists"
    verifynegative="verify this string does not exist"
    logrequest="yes"
    logresponse="yes"
    sleep="3"
/>

The interesting thing is that GUI for the tools shows response time graph that may be important for performance analysis of a system under test.

Mechanize

Mechanize is a Python library that mimics web browser based on Perl module WWW:Mechanize. Sample session source code:

br = mechanize.Browser()
br.open("http://www.example.com/")
# follow second link with element text matching regular expression
response1 = br.follow_link(text_regex=r"cheese\s*shop", nr=1)
assert br.viewing_html()
print br.title()
print response1.geturl()
print response1.info()  # headers
print response1.read()  # body

br.select_form(name="order")
# Browser passes through unknown attributes (including methods)
# to the selected HTMLForm.
br["cheeses"] = ["mozzarella", "caerphilly"]  # (the method here is __setitem__)
# Submit current form.  Browser calls .close() on the current response on
# navigation, so this closes response1
response2 = br.submit()

Sometimes, our application do heavily depends on JavaScript + DOM tree management to handle functionality, so we have to employ more advanced tools.

Basic JavaScript support

HtmlUnit

268HtmlUnit tries to understand JavaScript used on HTML pages to handle AJAX-based pages. It’s written in Java. Sample source code:

public class homePage {
  public static void main(String[] args) throws Exception {

    final WebClient webClient = new WebClient();
    final HtmlPage page = webClient.getPage("website name here");
    HtmlElement usrname = page.getElementByName("username");
    usrname.click();
    usrname.type("myusername");
    HtmlElement psswrd = page.getElementByName("password");
    psswrd.click();
    psswrd.type("mypassword");
    HtmlSelect select = (HtmlSelect) page.getElementById("cmbProducts");
    HtmlOption option = select.getOptionByValue("ITDirect");
    select.setSelectedAttribute(option, true);
    HtmlElement signin = page.getElementByName("SignIn");
    signin.click();
    System.out.println(page.getTitleText());
    webClient.closeAllWindows();
  }
}

HTTPUnit

HttpUnitHTTPUnit is a Java library that allows to emulate browser (has some JavaScript support as HtmlUnit). Sample source code:

    public void testGetForm() throws Exception {
        ServletRunner sr = new ServletRunner( "web.xml" );       // (1) use the web.xml file to define mappings
        ServletUnitClient client = sr.newClient();               // (2) create a client to invoke the application

        try {
            client.getResponse( "http://localhost/PoolEditor" ); // (3) invoke the servlet w/o authorization
            fail( "PoolEditor is not protected" );
        } catch (AuthorizationRequiredException e) {             // (4) verify that access is denied
        }

        client.setAuthorization( "aUser", "pool-admin" );        // (5) specify authorization and
        client.getResponse( "http://localhost/PoolEditor" );     //     invoke the servlet again
    }

Sometimes basic JavaScript implementation are not sufficient, so you have to employ full-featured browser to do the work.

In-browser solutions

XPCOM

mdn-logo-smXPCOM is an interface to Mozilla-based web browsers. You can control browser from many languages (Python, C++, …).

Greasemonkey

It’s a method to inject small piece of JavaScript to any web page. First tests have shown random driver using this tool is perfectly possible.

Java / Servlet probe based on web.xml / error-page tag

Now we have Java (J2EE Servlet based probe implementation) on board! Sample stacktrace from Java servlet:

event counter: 3
javax.servlet.ServletException: Exception details
at com.randomtest.test.ExceptionTest.doGet(ExceptionTest.java:15)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:324)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
at org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:654)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951)
at java.lang.Thread.run(Thread.java:679)

The installation is pretty simple (few lines in web.xml file + RandomTestErrorHandler.class on CLASSPTAH), you have to customize your server URL location:

    <!– randomtest infrastructure –>
<servlet>
<display-name>RandomTestErrorHandler</display-name>
<servlet-name>RandomTestErrorHandler</servlet-name>
<servlet-class>com.randomtest.RandomTestErrorHandler</servlet-class>
<init-param>
<param-name>RANDOMTEST_URL</param-name>
<param-value>http://localhost/randomtest-server.php</param-value>
</init-param>
<!– To show any config problems instantly –>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>RandomTestErrorHandler</servlet-name>
<url-pattern>/RandomTestErrorHandler</url-pattern>
</servlet-mapping>

<!– Error types to be handled by randomtest –>
<error-page>
<error-code>404</error-code>
<location>/RandomTestErrorHandler</location>
</error-page>
<error-page>
<exception-type>java.lang.Throwable</exception-type >
<location>/RandomTestErrorHandler</location>
</error-page>

 

PHP probe and server aggregated report added

Probe code for PHP has been prepared. Basic usage is:

putenv(“RANDOMTEST_URL”, “http://localhost/path/to/randomtest-server.php”);
require(“randomtest-probe.php”);

You can set those in common script included by every part of your application. With that any error, warning, notice are reported to the server immediately.

Additionally server uses local DB storage now to count frequency of stacktraces and shows them starting from most frequently visible, an example:

RandomTest.net report

event counter: 14
BEGIN RANDOMTEST EVENT process:/home/darek/public_html/randomtest.net/src/probes/c/test1.exe
sigsegv_handler
./test1.exe: MyClass2::myMethod2(char const*)+0×9
./test1.exe: MyClass1::myMethod1(int)+0×12
./test1.exe: main()+0×15
/lib/i386-linux-gnu/libc.so.6: __libc_start_main()+0xf3
./test1.exe() [0x8048561]: ??
RANDOMTEST EVENT END

event counter: 11
BEGIN RANDOMTEST EVENT
Undefined variable: undefined_var
/home/darek/public_html/randomtest.net/src/probes/php/test1.php:10 randomtest_error_handler
/home/darek/public_html/randomtest.net/src/probes/php/test1.php:20 fun_C
END RANDOMTEST EVENT

(…)

Note the server has collected stacktraces from two totally different environments: C++ standalone program and PHP script run on the server. That means you can track events effectively in big, heterogeneous systems.

The probe-server pair has right now the very minimal but useful functionality included. You can track your local development / QA team activities by catching every error / warning / notice that might occur during manual testing and might remain unnoticed. Full stacktrace included might help greatly with error localization and fix.

 

First basic probe for C/C++ environment

The source code for first C/C++ probe implementation has been just published. By using the following construct in your profile files (an example URL):

LD_PRELOAD=./randomtest-probe.so
export RANDOMTEST_URL=http://localhost/randomtest-server.php

You can collect crashes (SIGBUS/SIGSEGV) from any C/C++ application without using gdb, even without source code available. An example:

BEGIN RANDOMTEST EVENT process:./src/probes/c/test1.exe
sigsegv_handler
./test1.exe: MyClass2::myMethod2(char const*)+0×9
./test1.exe: MyClass1::myMethod1(int)+0×12
./test1.exe: main()+0×15
/lib/i386-linux-gnu/libc.so.6: __libc_start_main()+0xf3
./test1.exe() [0x8048561]: ??
RANDOMTEST EVENT END

Note that some C/C++ programs might need the following LFLAGS / CFLAGS:

  • –rdynamic to save symbolic names for standalone executables – linker
  • -funwind-tables to embed exceptions metadata even for C code – compiler
  • -exceptions for QT configure (if QT is used) to store unwind tables on a stack – configure

What’s Wrong With Automated Integration Tests?

A quite typical picture: software development company X, first delivery of project Y to testing team after few months of coding just failed because software is so buggy and crashes so often. Project Manager decides to invest some money in automated testing tool that will solve all stability problems with click-and-replay interface. Demos are very impressive. The tool was integrated and a bunch of tests were “clicked”.

After a month we have 10% of tests that are failing. 10% is not a big deal, we can live with them. After additional month 30% of tests fails because important screen design was changed and some tests cannot authorize them for some reason. Pressure for next delivery increases, chances to delegate some testers to fix failing test cases are smaller every week.

What are the final results of above tool?

  • unmaintained set of tests (and the tool itself) is abandoned
  • man-days lost for setting up test cases
  • $$ lost for the tool and training

Has the tool been introduced too late? Maybe wrong tool was selected?

In my opinion automation and integration tests don’t play well together. Let’s review then main enemies of automation in integration tests:

Initial state setup of environment and system itself

For unit-level tests you can easily control local environment by setting up mocks to isolate other parts of system. If you want test integration issues you have to face with integration-level complexity.

No more simple setups! If you want to setup state properly to get expected result you MUST explicitly set state of whole environment. Sometimes it’s just impossible or extremely complicated to set it using UI. If you set states improperly (or just accept existing state as a starting point) you will end with random result changes that will make tests useless.

Result retrieval after operation

OK, we scheduled action A and want to check if record was updated in DB or not. In order to do that some list view is opened and record is located using search. Then record is opened and we can add expectations to that screen.

OK, but if we want to check if “e-mail was sent”? We cannot see that in application UI. Catching on SMTP level will be too unreliable and slow (timings).

It will just not work smoothly.

What’s next then?

It’s easy to criticize everything without proposing (and implementing) efficient alternatives.

My vision of integration testing:

  • I’m not checking results for automated integration testing, it will just doesn’t work cannot be maintained efficiently
  • I’m usually generating random input to cover as much functionality as possible using high level interfaces (UI)
  • I depend on internal assertions/design by contract to catch problems during such random-based testing, they serve as an oracle (the more the results are better)