BlackBox Review

If you came here to read about the 80’s band singing ‘Ride On Time’ - you’re in the wrong place! 


This article is about Blackbox as a retrospective tool for understanding projects in your business, and client work. 


The name refers to the capture of information that can be used to understand where issues occur, and develop ways to navigate around these problems happening again. 


It’s commonly linked to the Aerospace industry’s use of the blackbox recorded. A point of information that was able to help even in the worst situations - which for them was loss of life. 


Whilst your business is not (hopefully) in a position where loss of life could result from your decisions. But incidents and projects can have a financial impact, or impact the business in ways that pose a threat. Loss of motivation from teams, budget overrunning, poor reputation from badly delivered projects.


You get my point. 


When it comes to retrospectives and blackbox reviews - I’ve found there’s more than one way to do this. So work out what works best for your needs over time.


It’s OK to adapt things to your business and people. If you work by yourself, you might not need the same level of input from others - but it helps to capture and store information for others as you grow and bring in other teams. 


Hindsight is a wonderful thing!


Use the blackbox review meetings to build up a knowledge bank over time. Putting in place small gains and improvements to your projects and processes will help your business now and in the future.


Planning your Blackbox meeting: 


Keeping things in time bound, but making sure you’ve got enough time to fulfil the review properly. 

An example of this is look at how many categories/topics your review is going to cover. 


Have a moderator, and a time person!


The moderator is someone that everyone in the meeting acknowledges will be the one to move people on from their point conversations start to move away from the review questions. 


Usually, whoever is the project manager for what’s being reviewed will be the best person usually. Or you can empower others to learn to run these by swopping or having a rolling order. You might have someone you nominate to run them all as they are not tied to any outcome and can manage the meeting with no hidden agenda. 


It’s ok to try a mix of things until you choose what works most naturally for you and your team. 


TIP: You can randomise the time keeper in face-to-face meetings by marking any snacks (in packets) that you’ve put out. Hand round a bowl of snacks and make sure there’s one that’s marked or is the only brand type of that snack. Whoever picks this is the time keeper. 


Getting the timing right on blackbox meetings and reviews: 


If there’s a small group of 5 people, each point might need each person’s perspective across the three areas below. Then the accountability agreements. So that means each person could speak for 3 minutes on each topic. Plus some group conversations. 


  • If I have a meeting kick off, and house rules talking for 5 mins total ….

  • Then 5 people, across 5 topics = 15 mins per person ….

  • After that an accountability group conversations of 10 mins total ….

  • Finally a round up summary by the moderator ….

  • My timings would need to be a minimum of 1 hour 30 mins for a blackbox meeting. 


The bigger the team or the more points you want to explore may require longer meeting times, or split into two meetings. But it’s often easier to keep everyone focused in one meeting, than two!


Ask for advance feedback before your blackbox review meeting:


You can ask people to add their thoughts into a survey - pull this together and work through the points in order. Again - timing will be easier to work out on the review time, but people are just summarising their points from the detail they’ve already given. 


Or the moderator can talk through the key findings and bring this to a shorter meeting time by focusing on the accountability. 


Think about when a good time for a review or project and work is. If not every stakeholder can make the meeting, you might want to give yourself a regular window or planning any blackbox within a 30 day period of key work being delivered. 


Using a Blackbox Review Template document: 


Here is some context on the key areas you’ll need to capture on the template I’ve created to support your review with. 


You can choose to have each person go round the group and talk, or you can do what we did in the recorded session and ask each person as part of the same topic point. This is something I personally prefer as people will often acknowledge if someone’s already captured what they were thinking - and it can speed up the process. 


You need to think about where you will store your completed review docs - and how to name them (naming conventions)  so it’s easier to manage as you build up more over time. But that’s part of your internal process development. 


blackbox image 1.png



You can break up your projects and reviews into smaller sections or Categories and Topics. This can help to show if there's lots of improvements to one particular part of your team, or processes in the projects. 


What went well? 

It’s good to focus on the positives first. It gets out of a blame game mindset. 


What can we celebrate or acknowledge as going as planned or better than planned? What doesn’t need fixing, what’s an asset to future work? 

Did your systems and team members work well together?

Was there any direct customer feedback that was captured?


What could be improved?

Capture any blockers to your delivery of work, any single point of failure people in the team, or things that negatively impacted the process and delivery of this project. Don't just look to assign blame, but be eager to self-identify if you felt you could do something better next time. 

Remember, you are looking to not isolate people in the team, as this will make reviews harder to get honest feedback from for learning. If there's issues that need support outside of this meeting - put these into reviews, or 1-2-1 settings, or follow your HR processes. 


What actions need to be taken before the next project?

List out the ideas or practical steps you can take to make changes or improvements. Don’t forget if there’s positive aspects to also include these.


Did someone do really well in their execution - does this mean you could explore offering incentives or rewards for ‘Person of the project’. It’s not just the negative we’re looking to build forward upon. 


If there’s no easy answer, you can put in here to explore and research options. But this will need the same accountability below as steps and process changes would get. 


Accountability for blackbox learning:


Accountability for making improvements breaks down into 3 key areas. 


The first is to prioritise what needs to be done first. Just because you can take action doesn’t mean you should do everything.


If you’re tight on time, or people to implement the changes you can review your tasks and further prioritise these so you know what needs to be done first. 


You can prioritise them into processable actions using either of these reminders:


  • PIE - Potential, Impact, Effort. 

  • ICE - Impact, complexity, Effort.


Create a scorecard number or scale for the work to be done. For Example. 

  • Potential = 1-5 (highest potential would be 5)

  • Impact = 1-5 (highest impact would be 5)

  • Effort = 1-5  (You might want to reverse the logic here, and put the easiest to do at 5)


This model would mean the higher the score something has - the more YOU SHOULD do it!


If you kept ease of effort at 1 for easiest this could skew your scoring and make it harder to know which should be done first. 


Logically then, low score would be low potential, low impact, and require more effort. 


Updating email templates might be  (P = 3, I = 4, E = 5) - Total would be 12 out of 15. 


Secondly, decide on who’s going to take the action on.

If you work for yourself or are a small team, some blackbox reviews will mean one person will take the lion’s share of the work to make the changes happen. Where possible try to spread this, or at least acknowledge that they will work through in priority order - and might need support or other work moving off their shoulders until it’s completed. 


Thirdly, decide on a time when this needs to be done by.

Knowing what needs to be done first is great, but not having any way to plan when these changes will be made renders the time spent on the process so far wasted. Putting realistic completed dates, times and deadlines in here will increase the likelihood of these changes being made. 


TIP: Whoever is likely to be doing the work on any changes will need to be in the meeting. You can’t nominate people to have things done if they have no input on what needs to be done, by when. Deadlines will be missed!


Once the meeting is finished: 


  1. Store your doc in the place you’re keeping these, and share the link out to the people so they can find and work through their planned updates (to the dates agreed). 

  2. Book in a follow up meeting - in about 30 days from now - so you can double check what’s been done, and by whom. This will encourage everyone to work to put the changes in place. Otherwise you risk losing track of things not getting done. 

  3. Ask for feedback on how to make the next black box review better! It’s a learning process which will iterate over time. 

Recommended reading. 

Black Box thinking book by Mathew Syed