Tuesday, 27 October 2015

A single source of testing truth...

Truth. Oscar Wilde said it best I think:
'The truth is rarely pure and never simple.'

In terms of vehement debates at the recent MEWT gathering in Nottingham, probably the talk which generated the most feedback and opinion was Duncan Nisbet's ‘The Single Source of Truth is a Lie.’ To be honest I was relatively quiet during the debate, as it was straight after my talk and I also need time to parse such things, hence this blog. 

A link to the slides can be found here:


What Duncan said in my head…

Those were the slides, this is how I understood the talk given by Duncan. First up, there was an admission by Duncan he was just putting this one out there for feedback, which is kind of the point of MEWT really. Second up, there was belt and braces, a definition of truth:
‘Conformity with reality or fact’ or can be otherwise known as ‘verity’
Thus began the front loading of the mind with terms with multiple, deeper meanings depending on their context. To be fair to Duncan, when he picks a subject, he doesn’t dance around on the edges. Truth. 

He used the ‘Three Amigos’ device as a way to introduce the topic, namely how can a session such as that generate a single source of truth, a shared understanding that can be taken and built. This might be living documentation (a la Specification by Example), which is semantically similar to defining some acceptance tests to drive the development. However it manifests itself, is it possible to define truth for a given feature/function/context/situation?

I believe the gist of the talk and the debate was a loose consensus that truth is a multi-faceted beast. Duncan credited James Bach with the following (the three pillars below are littered throughout many disciplines (social work, medicine, mental health) as a model in their own right). The truth is made up of (as I understood):

  • Social truth – conformity with realities commonly held within/across team/social strata;
  • Psychological truth – conformity with realities held individually within one’s psyche;
  • Physical truth – conformity with the reality presented by a physical artifact, such as your product.

Models of truth on top of models for testing…

Duncan then overlaid this on top of the Heuristic Test Strategy Model:

  • Social Truth ↔ Project Characteristics – linked by a need for shared understanding about the feature/function/context/situation;
  • Psychological Truth ↔ Quality Characteristics – linked by how one might feel about the feature/function/context/situation;
  • Physical Truth ↔ Product Characteristics – linked by the production of artifacts pertaining to the feature/function/context/situation.

This resonated with me, I could mentally link this version of how the truth might be determined with a model of questions. Which, for myself, is the point, in that having a model which allows you to determine what you might consider to be truth in your context, is rather useful, even if the precision of that determination is fallible.

The debate began afterwards, ranged from definitions of premise and assumption, disappeared down etymological rabbit holes. I think eventually the white flag was waved and we moved on.

Should testers even talk about truth…?

What do I think of this debate then? I will keep it short and simple, in the light of the potential maze this represents. As a point of order, and in concert with the model presented by Duncan, most of what follows discusses physical truth, namely an artifact/product and what it might do. Social and psychological truths deserve tomes of their own.
When the word ‘truth’ is used in a testing context, I generally think of a few things:

  • Words that we, as testers, shouldn’t use. I would probably put ‘truth’ into a similar bucket as ‘full’, ‘complete’ or ‘done.’ You utter these terms and the ground beneath your feet becomes decidedly shaky. Not one I would put in my safety language locker. Mainly because these terms are literally taken sometimes and truth (to some) seems so darned final.
  • We are in the information business, over the decision making business. “This is what it does” may be more of a tester’s domain over “This is what it should do.” After all, never the twain shall quite meet in beautiful clarity. That is not to say a blend of the two is not something to strive for (preferring early test involvement over at the end, ‘as a service’), but we should be mindful of our core principles.
  • Hang on. Have we not already got an approach for this, as in identifying oracles and being aware of their fallibility? Maybe we’ve already done this question. Nothing wrong with revisiting the Oracle Problem, but I believe that approach remains fundamentally sound, and leaves room for context, whereas truth chases the absolute (doffs cap to John Stevenson here, but I subscribe).

Is this (yet) another impossi-task…?

Speaking of absolute, is truth another techni-coloured dream coat wearing, rainbow generating unicorn with diamonds for eyes that we seem to continuously chase in software development? It sounds suspiciously like that process of nailing down ‘stuff.’ Truth seems to me to be subject to change like all other things, and the more we try and pin the blancmange of truth to the wall, the slipperier the world gets. 

We (in software development, and those whose business depend on it) seem to rather enjoy setting ourselves impossible, contrary goals (deliver this huge thing that the world will still want in two years’ time for example) which directly grind the gears of the world. Maybe this is just one of those. We’ll get over it one day.

Certainly made me think. Truth might just be a journey, and not a destination.

Sunday, 18 October 2015

My First MEWT...

Thanks to Wikimedia Commons for the image
A few months ago I got a very intriguing invite from a certain Richard Bradshaw to contribute to MEWT, an event I had been aware of out of the corner of my eye for a couple of years. The event is held at the beautiful Attenborough Nature Reserve and we delivered our reports within the Media Centre, perched in the middle of the lake. A stunning venue, and a great setting for learning.

As well as being my first MEWT, it was also my first peer conference, where experience reports are presented and then the floor is opened to questions, clarifications and comments. After the floor was opened to determine the running order, we took a vote. I'm not going to lie, I had a hangover, after discussing what the time "half eleven" means to a person from The Netherlands into the relatively small hours the night before. This naturally mean't I would be first up. Of course it did.

So I began to talk through my model for surfacing unrecognised internal models, inspired by a number of coachees who spoke subconsciously of their models, struggled to articulate them, and applied them unwittingly. To be honest, this was quite a nervous time. This model had not seen the light of day outside my brain and that of a few of my coachees. It is very personal, like anything one has created, and to expose to scrutiny can be painful. It was not however. Instead I received thoughtful feedback on potential improvements, also some of my less convincing answers prompted me to re-examine my own thinking on some aspects of the model. Areas of feedback which really interested me:
  • Being careful with goals - goals can drive behaviours, perhaps not in the way you intend.
  • Having a step to revisit goals on an iterative basis is valuable, as the world changes around the coach and coachee.
  • Sharing between coachees - all my coachees are on this path, so why not encourage them to share with each other, giving shared learning opportunities and empathy with the journey of others.
  • To visualise the model in some way, as opposed to the mindmap I had. Coaching ebbs and flows, so I think a means of communicating the model in this manner would be valuable.
MEWT has added the following to my blog post list:
  • Testers talking about truth - inspired by Dunc Nisbet, although I will need to take a week off to investigate, parse and articulate this one!
  • Testers improving themselves/awakening to a more intentional, thinking approach - inspired by Ard Kramer and Geir Gulbrandsen - at what point do we wake up and no longer apply rote models of testing to all problems? I know when I did, I hope to explore this further.
I could do them all as all the ideas presented certainly made me think. Maybe someday, but I'll start with those two. All that remains is to thank everyone. Those who spoke, questioned, organised, facilitated, tweeted, discussed and all the other activities that made MEWT 2015 a massive success.

Also see:




http://www.associationforsoftwaretesting.org/ - who were hugely gracious in their sponsorship of the venue for the event.

Thanks to John Stevenson for this great photo. Tutu's have been removed to protect the innocent.

Saturday, 3 October 2015

Leeds Testing Community unConference

A few of you will know that we (well, Stephen Mounsey, Nick Judge, Fredrik Seiness, Phil Hargreaves et al did all the hard work, I just flounced in and presented a workshop) have recently given birth to the Leeds Testing Community unConference. All conferences start from an acorn, a twinkle in the eye, and this was no exception.  I didn’t want to let it pass without blogging on it, as I believe it to be the beginning of something big! There is a real thirst for this kind of event in Leeds, a thriving tech city with loads going on.

A quick whistle-stop tour of my highlights:
  • Uno – Laurence Wood presented on his agile heroes, including my close testing and monitoring pal Gwen Diagram. I will not say the ‘D’ word. Also, on one to one ratios of developers to product owners (pinch me), a very strong start from a great speaker.
  • Dos – My mobile testing workshop, entitled ‘Extending your testing senses.’ Despite being the only person currently testing in a mobile context in the room, everyone really got stuck in to using the on device Android Developer Tools. Testing search functions on popular networks using CPU monitoring, layout tools and many more. I heard a lot of heartening ‘I didn’t even know my phone did that’ type of comments. Even better the same tools were used in the afternoon workshop. Joy.
  • Tres – Stephen Mounsey and sketch noting was a great interactive session, I felt a sense of accomplishment and satisfaction as my barely legible doodles became a coherent map of the session. Or something like that. The key learning was turning previously dreaded meetings into something engaging and being present. With a tangible output at the end of it.
  • Cuatro – An honourable mention for my good friend Clem Pickering, whose presentation of the Palchinsky Principles really resonated with me, with strong threads of experimentation and viewing failure as learning. Slide of the day was a surprising number on how Prince Charles and Ozzy Osbourne share a surprising number of characteristics, showing just how much perspective impacts your assumptions while testing.

All that remains is a massive thank you to the organisers, hosts (Callcredit Information Group, who I can testify are an extremely engaging organisation for those in software development) and my fellow speakers. 

Isn’t it exciting to be in at the beginning of something awesome?

Saturday, 19 September 2015


The world is really complex. Layers upon layers of stuff exist in the systems we test. Old stuff on top of new stuff, augmented stuff, rewritten stuff, interconnected stuff, internal stuff talks to external stuff, data from here, there and everywhere. Stuff is done the move, stuff is done from places we never imagined. We are stuffed with stuff, and stuff about stuff. Metastuff.

The Problem with Stuff…

When testing this stuff, it’s pretty easy to get quite full of metastuff quite quickly. Then you get that post-Christmas lunch drowsiness, where you probably should be playing a board game with your family but for some reason you can’t move and are staring at the EastEnders Christmas Special (Phil is leaving Sharon, Beppe is selling E20). Being stuffed with metastuff has left you dull-witted, unable to focus on what matters.

Have I seen this in my fellow testers? Yes. Have I experienced this when testing complex stuff with multi-layered architectures? Oh yes.


There is a way to cope though. You can do it. You remember when the Emperor shoots lightning from his fingers in Star Wars? You need to be able to do that. In reverse, where your fingertips are your enhanced and extended senses, and focal points for targeted information, filtering out the stuff. You can cackle like a maniac while you’re doing it if you want.

We need to complement our existing senses. This can be achieved by creating (and then sharpening) our feedback environment while we test.

First Principles…

As is healthy in testing, as it is in life, let’s start with principles:
  •  Find out what matters to your stakeholders. If you have one client who provides 90% of the revenue and 90% of what they do is one specific action. Focus your feedback there. I am saying this for completeness as you already know this right?
  • Complex systems are best understood in layers, receiving feedback from each layer is critical to creating an effective feedback environment.
  • Do not fear asking for systems to return concise information pertinent to testing, timestamps, exception information. This includes the reduction of noise within a logs and auditing tables, so only critical information is presented, not just everything for information's sake. Verbosity can be controlled, your early environments can be more so than your production instances.
  • Distinguish between synchronous and asynchronous parts of a system. Knowledge is power. If you have a system which does some batch operations synchronously, while other asynchronous actions are occurring, you need to test that, but also a way to isolate the feedback from either.
  • Use tools and techniques to make the feedback environment interactive and colourful. We see patterns, it’s one of our strengths, but can be assisted. Make it move and make stuff that is out of the ordinary look out of the ordinary.
  • Knowing how to set your environment to a known state is critical, not just test data either. Being able to actually set your system state to a known state is critical to a feedback environment. Which is clean and as predictable as you can make it.

Specification by Example…

Here comes an example setup for a system with a three tier architecture which I find is a tremendous way to show what I’m talking about. For example, this may be the setup for a testing session:

Establishing a baseline…a clean, (relatively) known system starting point

Setting up to listen while you test…a way to generate continuous feedback

Gathering feedback and generating coherent information…relevant and timely to the session

This shows the visual steps of digging deeper. The arrows denote the level of interrogation. As in find an error in the php logs, discover the Zend Event and following Code Trace.  This depth of information can be further refined by targeting log files with specific criteria based on what you are focusing on by piping log files into grep for example:

Looking out for specific process messages using tail –f /var/log/messages | grep “sftp” or monitoring processes similarly with watch –nd 10 ps aux | grep “httpd.”

The above is a good link into tooling, which is less the focus of this blog, but when used complementary to you own senses and configured intentionally (as in not to swamp one with information and pointing at the focus of the testing session) hugely useful. Personally I think that is a separate blog, and it is probably useful to show how I leverage certain tools (such as Android Studio for example) to enhance my testing senses.

Debugging innit…?

Onwards. This is distinct from debugging in my view as you are providing the where and what of a problem, over the how and why. There is no barrier to how and why of course, depends on your context. This of course depends on a certain level of technical awareness (as in I know the strengths and weaknesses of technology and how to use it to enhance the information I glean) but in a multi-skilled team this should not be beyond a collaborative effort. Further to that, I do believe that a feedback environment such as that above is a team artifact, not only a tester’s domain. A testers bug reports should complement the debugging flow in a given technology or team, so only a few steps are required to begin the diagnosis of the how/why.

Further to this, your feedback environment might look a lot like your monitoring setup which can be applied on a micro scale (for a tester to use for example) or a macro scale (for a system, cluster or federation to use), thus creating a link between common feedback mechanisms for that tricky transition from development to operations. I won’t say the buzzword.

Over to you…

Next time you begin a testing session, ask the question:

Are my stakeholders getting value from what information this system can provide?

Then go a step further and ask:

Is the information the system provides filtered to my focus and therefore meaningful in context?

This will not replace your (un)common sense, but it may help save you from being inundated with a growing tidal wave of stuff. And all the stuff about that stuff.

Wednesday, 9 September 2015

Under Pressure

As a tester, you might recognise this feeling.

The world wants your tests to “pass.” Developers, Product Owners, Project Managers, Executives, everyone is looking at you, waiting for you, wanting the tests to “pass.” Wanting this feature to be delivered, this final piece of the going live puzzle.

Has it “passed” then?

Whatever it is, it hasn’t, it can’t, at the heart of the matter, that isn't how exploration works. There are perceived and non-perceived problems, inconsistencies, misunderstandings, and conflicting stakeholder perceptions. It’s not your judgement to give, however this doesn’t mean that you don’t feel any pressure as a human being.

When you feel it, here are some fun thoughts that you might want to bear in mind:
  • It’s really not your pressure. One of the key lessons I have learnt is not to accept the pressure of others. I’m a real responsibility magnet (there is a whole other blog there). But I make no promises when it comes to testing, I just mine for information. Stop accepting others pressure and watch your testing life transform.
  • Shit ain’t all down to you. It’s not honest, you are special and important but… For example, very early in my career, I had the last blocking bug on a gigantic, multi-multi-multi million pound offshoring project. I was being crushed by ownership of this hideous carbuncle of a bug. Afterwards I realised, it wasn’t mine. The root of that glittering pearl of a problem was way, way back in the mists of the project. Somewhere I had no control of at all. In another organisation. In the past. I had zero influence, literally nothing to be done.
  • It's quite popular to say “I’m giving information about quality to someone who matters.” Don’t even say that. Just say information to someone who matters. We need more distance from quality, not less. Every time you talk about testing and quality, you are creating a false link. It’s not real and confuses those who are already confused about what testing is for. Stop it.
  • Lastly the Zen bit. Nobody wants tests to fail, probably. But this is the rub. They don’t fail. You just learn. It’s true. Once you change from failing to learning, an infinite dimension of satisfaction opens up in your testing. You will find serenity in this thought, even in the most turbulent organisation.

Just remember, as a tester, the moment you allow the words “the test has passed” to pass your lips, turned anything green, given a thumbs up, cheeky grin, however you do it, remember the myth you are perpetuating. That the information gained from exploration is a tester’s responsibility.

It’s everyone’s responsibility. I hear that a lot. Let’s live it.

Saturday, 13 June 2015


I’m currently experiencing something I never thought I would.

The technology team I work in isn’t the slightly odd, dysfunctional part of the business, tucked away in the corner, showing signs of madness, gibbering binary nonsense at anyone who strays within range. I believe we are a high functioning team and have gone further, we are reaching out and contributing to improving the wider organisational system. I know, I have fallen off my chair several times.

How are we achieving this state I hear you ask? I won’t provide an illusion of perfection but I think we are doing these things really well:
  • Saying yes, because technology is the art of the possible. Genuinely living the ethos that we can build what you need, share your priorities, help us understand, then we’ll get to it. Go to point 2.
  • Releasing stuff regularly to get feedback. I know this is old hat, but you’d be amazed what it can achieve. Roughly weekly, mostly when it’s valuable, not overloading. Asking if you want more stuff, not here’s more stuff.
  • Helping to determine what the organisation considers valuable over falling down the rabbit hole of what we can provide. The perennial conversation that we’ve all been involved with, I’ll tell you what I want, when you tell me what you can give me. Which usually results in the wrong solution to a poorly understood problem.
  • Providing coaching on the implications of technology. As in technology and what it means for the business proposition, not just how technology has been implemented. Where is it strong or weak? What choices did we make? 
  • Espousing the value of team. That every team should have everything it needs in order to succeed. If the team needs these skills, then let’s make that happen. Also, not just filling gaps. For example, not just hiring testers, but encouraging testers to encourage others to take responsibility in appropriate ways. If you hire more testers, you’ll more than likely get more testing, but less likely a great deal more value.
  • Genuinely hiring for behaviours. I’ve played at this before, but that urge to think ‘commercially’ has meant rushed decisions with months of pain. Making the hiring process equitable (you have to be happy with us, not just us happy with you) is much more pragmatic than the misery for both parties of being a square peg in a round hole.
  • Asking for clarity on the mission. Usually the technology teams are the last to know and/or understand the mission of the organisation. Not us. Rather than trying to hide on the periphery, we are front and centre of the mission, discussing how technology can enable us to realise that mission.
I personally love being a part of all this. I hope we can create an organisation which genuinely lives its values. Something I’ve been seeking for a long time.

Sunday, 17 May 2015

Weekend Testing Europe - Testing Session for LinkedIn

Great time had during Weekend Testing Europe, recommended for all to sharpen up your skills, ask questions, or just get a bit of practice. The exercise in principle was compare features from mobile to desktop on the beautiful LinkedIn app/site...

For those interested my testing session files generated:


Linked In Mobile App

iPhone 5s iOS 8.3

Search Functionality

## No access to search algorithm to verify results


##User needs to authenticate for search function to be available.


##On tapping search and entering no details, contacts appear alphabetically ascending.
##Cancelling returns to home tab and reloads the Newsfeed.
##Options for the user:
Contacts appear alphabetically ascending.
Displays a set of jobs you may be interested in 
? Unknown algorithm for this, would need to be verified ?
Displays option to set location
In a new dialog you can search for a location or use current
Using current prompts for access to location services
Don't allow and the screen goes blank you can only cancel.
App remembers that you have disallowed location use.
Displays the companies you are currently following
? What happens when you follow no companies ?
Displays your currently joined groups
? What happens when you have no groups ?
## Search
Adding search terms filters your own contacts first, then returns those you may know
? Unverified algo here ?
TO CHECK - Searching on a full name of a contact you already have returns those you don't only
Can search with only 1 char -  hopefully this is managed in the search algo
TO CHECK - Cannot search by company, only title
Title appears to be fuzzily matched, without location set you get jobs from all over.
With location set:
Latest jobs are returned first, lots of jobs in Leeds.
Location can be removed, returns to previous results.
Job title and location can be used in conjunction. 


LinkedIn Desktop

Windows 8.1, Chrome 42

Search Functionality

Again, no understanding of the search algo
Should be aware that the aim is different here than the mobile experience, search for power and results rather than speed.

User does not need to be authenticated to search
Minor functionality is available, searching by first name and last name
The search is a true 'elastic search' rather than the linear, categorised search.

Entering text exposes:
Showcased Pages
Can search with 1 char - 18 results are filtered each time.
The 18 results are shared between the above categories
The search dynamically filters as you type.
Searching on a full name reliabily returns connected and unconnected users  (mobile app only unconnected users)
Company search defaults three options
Those who work there
Those who no longer work there
Typing a search terms and the clicking search opens a new page with advanced search options



-Desktop was a true elastic search, built for power with a dependable connection, mobile much more category dependant and about returning immediate results
-Desktop was much more powerful, but both filtered and gave suggestions as you entered data. Mobile good for filtering what you already have, desktop better for finding new stuff.


If you haven't already, get yourselves signed up for next time: