Client: PepsiCo – (In-House SaaS Product)
Type: Desktop, Mobile, Tablet & HoloLens Mixed Reality Application
To create an environmental monitoring solution to increase pathogen result transparency in the food manufacturing factory workplace. Linking to sanitation systems to enable simple communication between different data sets. To increase front-line factory workers efficiency by guiding them to sample locations and to improve general sampling practice and digital training. Seek and destroy pathogens, to reduce risk and report areas of poor hygiene that need attention, with risk assessments/corrective action – for auditing purposes.
To create an automated task based LIMS (Laboratory Information Management System) to allow full transparency of pathogen results on various locations within the factory. To show CAD map data for easy training and to show a data rich audit trail for pathogen environmental monitoring. Schedules and sample locations are set up via the Desktop and tasklists are delivered to frontline workers via a HoloLens 2 headset or Mobile devices. Desktop API integration with a 3rd party laboratory tester for realtime results that automatically triggers corrective action plan tasks, for positive results.
A design thinking workshop was setup to explore users problems & needs – with the intention to find user pain points and issues that need attention. Understanding the users motivations and frustrations was important to begin the planning stage of the MVP (Minimum Viable Product) to provide the most customer and business value.
Lots of key research data was gathered during the workshop. The Design Thinking research workshop was formatted into the following key areas:
After the Design Thinking workshop it was arranged that I shadowed the Laboratory Technicians during the daily swabbing/sampling process. Using note taking, data gathering and video footage of the entire end to end processes allowed us to research the current procedures and uncover any weaknesses or potential improvements that could be made, from a digital transformation perspective. This allowed us to move to a paperless Azure cloud based digital solution that could be shared across multiple locations across the world.
After the design thinking workshop ‘Must Have’ elements where put together for information architecture. This helped form a clear direction as to the most important features and functions that the desktop needed to perform. Essentially the current swabbing process involved lots of paperwork and reliant on a 3rd party lab testing company to supply the results. The proposed desktop application was going to use an API to supply real-time testing results into the desktop from the testing company (EuroFins). This allowed for complete transparency for any positive results that where found in factory.
Wireframes for the desktop where quickly mocked up to create a 2D clickable prototype, this helped us inform a basic skeleton structure of the product, to get a feel for all the UI components and usability elements that would be needed. Adobe XD and Figma were used to quickly explore a working clickable prototype that was shared with the team and stakeholders for feedback. Both XD and Figma software packages use vector based graphics to build the foundational UI elements, this makes for an easier way to create a Design System to be scaled and shared across different platforms for the traXR product.
A quick 2D clickable prototype was created, some quick usability testing was done to see how coherent the design was. For the purposes of the prototype no grids or columns were used at this stage and there wasn’t an introduction to colour, as can be seen from the animated GIF above.
After some design iterations were made on the Low-Fi prototype – a Mid-Fi prototype was produced, now the columns, margins and gutters were put into place to give a nice aesthetic layout. Different shades of grey were used to help show a hierarchy of information. The optimal display size ratio is 16:9.
A Design System started to be incorporated into the prototype. The foundations of Material design was used with the basic principles of interactions and animations to allow for a beautiful cross platform experience. Using components in Figma meant that the layout and UI components could easily be linked to master components. This sped up the production pipeline especially when shared graphical assets are used between different platforms.
The Mid-Fi prototype was used for usability testing to make sure all elements of the design were easily understandable to the user, with a natural cognitive hierarchy for the navigation menu and displayed components. Iterations and design changes were made to the prototype as we progressed with the products development. Once these designs were initially signed off the developers could begin planning for the foundational programmatic approach and which frameworks would be most appropriate to implement, for the success of the product.
Again, with further iterations and adjustments on the Mid-Fidelity prototype this led to creating a primary and secondary colour palette with gradients, images, visual alerts etc. along with the typeface options to remain consistent throughout the traXR products. Following on from the TraXR branding guidelines a Design System was beginning to be integrated into the Desktop product. The colour palettes, typography and UI components where going to be used across different platforms where applicable. The two other platforms were the HoloLens 2 and Mobile (iOS and Android).
A final beautiful clickable prototype was created to allow us to create some reliable usability testing, after several iterations the final aesthetics and user experience was adjusted to suit the users and business requirements. Where applicable all the animations and interactions were shown in the Figma prototype to give an pixel perfect representation of the final product before it was built. The Figma files were shared with the developers so they could extract code snippets, icons, font sizes, pixel spacing, margins, padding etc with ease. If there was any issues, the problems could be raised and dealt with efficiently to speed up production time.
Once the colour palette and font were selected for the Design System, these primary and secondary colours were used throughout the rest of the different platforms (HoloLens and Mobile applications) this allowed for design consistency throughout all the platforms.
An outline icon set was designed for easy recognition and integrated the material design guidelines for the creation of these icons. Most of the icon set was accompanied by text onscreen, to help guide the user as icons alone can be very confusing for first time users of any product.
We adopted Microsoft Azure cloud with data brick integration, this helped us build the AI and Data Predicting trends with patterns. Using Power BI to setup and drive the data sets helped higher management look at trends and easily access the efficiency of staff members, number of positive results over time, pathogen types and outbreaks, effect on air humidity with outbreaks etc. Using Power BI also enabled us to integrate data with tableau, which was used for analytical data by the client. In some cases, we used sensors located in the factory to help display different data sets to predict patterns and trends that could occur. This enabled problems to be fixed before they occur, this helped save the clients line production downtime, which intern saved the factory money and product recall.
Once the sketches were reviewed it was a good stepping stone to prioritise what interface elements would be needed to communicate to the user, for their ideal journey.
The main function of the HoloLens was to aid the user to complete sampling tasks based in factory locations. Using Azure spatial anchors the location data was then used to form patterns, heat maps and trends based on all the data gathered from the user. Using spatial anchors allowed the system to pin point the sampling task location within a 1 meter accuracy.
Other features which were prototyped for the proof of concept:
After plenty of sketching ideas and task flows, I wanted to understand a 3D spatial interface and how we would tackle the problem ahead. It was clear at this point for the MVP deliverable the HoloLens apps main functions would be completing a sampling task and training user to take a sample safely and in a controlled environment.
After extensive user research and requirements, the Hololens 2 is used as a data collection tool, its main user requirements for the sampling tasks were:
A quick 2D clickable prototype was mocked up for the flow of the HoloLens application, the real challenge was the limited field of view, navigating gestures in the headset and to make the UX aesthetically clean and easily understandable.
Marvel app helped show developers and stakeholders the linear cinematic type flow of the task needed to be completed by the frontline factory worker. A simple card sorting exercise was used in order to establish a UI hierarchy. This helped identify a hierarchy of key UI components and how we were going to display them to the user. The black screens are the same aspect ratio 16:9 as this is the FOV in the Hololens, this showed the constraints and limitations of what I was designing for.
After initial prototyping and experimenting a circular UI animation system was expanded on as this felt natural and organic from an aesthetics stance.
The main feature from the initial prototype was to display all the tasks as task cards in the HoloLens, this enabled the user to quickly scroll through their task list and select the task to be done. The HoloLens also used the 3d spatial anchors to locate the user and display the nearest task to be done, based on a proximity. Using subtle wayfinding arrows the user was also guided to the task location in the shortest and safest route.
Using vector graphics in Figma allowed me to quickly create a 2D useable prototype to establish the user flow for each task to be completed, this really helped convey the 2D prototype into a 3D scene, and gave the developers the concept of what needed to be achieved.
There were various task types to be completed these consisted of:
Each of these task types has different statuses throughout the task lifecycle:
Throughout the journey of taking the samples in the factory the desktop displayed to the user the exact status of the task at any given time, to give the end user complete clarity of the task status.
There is a pre-sample setup this involves placing barcoded stickers onto plastic zip tie bags, the correct number of bags and samples is shown on the desktop for each daily sampling routine task.
Once the user is in the correct area, the user is then encouraged to look around their environment and scan for any potential dangers or dirty areas within the factory. Once the user has chosen an area then a 12cm x 12cm grid is overlayed and pinned, this is the location of the swab. The user is then encouraged to take sample, scan barcode and then return the swab into the post swab bag. The user then has to take at least 2 photos of the area sampled one close up and one distance to give context of the swab location, they can take more if needed. The photos are time and location stamped and then viewable on the desktop application.
The user is not required to leave an audio recording but they can add a up to 2 minute audio recording, which can be transcribed using Azures voice to text service, all audio & photo files are linked with each of the different task types that can be taken in the factory. The Photos and Audio files can be edited before each task is submitted to the cloud to eliminate errors.
Once the samples are taken and ready, they are then sent to the testing lab, each sample is booked for collection through the desktop app. This is automatically generated through an API, which automatically raises a PO and notifies the testing lab that the samples are ready for collection. As soon as the samples are tested in the lab, if there are any positive results, the related tasks are instantly flagged in the desktop and a risk assessment task is then automatically created.
The risk assessment task is classified as a high priority task and depending on the location and pathogen type tested an emergency response is required by the team. Once all the risk assessment criteria has been assessed then a starburst task is created and linked to the initial positive result, this keeps a quick and easy reference point for the stages of the starburst task and all elements of the tasks are traceable.
When a Risk Assessment task is active the Starburst task is then created, there are 3 levels of the starburst task:
These are encouraged to be taken as soon as possible to find the source of the outbreak. The starburst tasks consist of taking multiple samples at 1 meter spacing up to a 5 meter radius from the initial positive sample. While completing a Starburst task concentric circles are visible in the HoloLens to show the distances of all the samples this enables any pathogen outbreaks to be traced and logged in a transparent way.
From the development of the HoloLens is was clear that the traXR mobile application was going to contain the following main features:
From a digital strategy perspective, the mobile phone offers the basic functionality and features of the HoloLens and is a cheaper solution that some factories requested during the researching stage of the product. The mobile application obviously doesn’t have the same handsfree functionality as the HoloLens and there are limitations of the mobile device.
The same task user flow was used from the HoloLens concept it was just initially prototyped out with basic wireframes. We had to be mindful of the UI transitions between a 2d flat screen then into the camera environment for the augmented reality to work seamlessly.
The HoloLens has a 3D spatial camera that builds a 3D mesh when scanning this technology isn’t mature enough to use with a standard phone at the time of developing this. We opted to use feature points to pin onto, while we took the samples at the location. Feature points work using high contrast points on a plane to create a surface, after extensive prototyping this was the most favourable solution. It wasn’t ideal, compared to the HoloLens 3D spatial anchors technology with the camera.
After initial concepting the task list was the most important part of the application as all of the current daily and prioritised tasks. A quick and dirty prototype was put together using Figma, this allowed for quick user testing to validate users’ decisions and thought process when using the mobile phone.
Once some iterations and product decisions were made with the Low–Fi Prototype this led onto the final Hi-Fi prototype which mirrored the HoloLens in basic functionality but on a mobile device.
It was critical to try and hit 60FPS with the HoloLens app as when complex shaders and lots of geometry where introduced there where significant performance issues. When the frame rate of the HoloLens app drops under 40 FPS there was noticeable latency which led to nausea for the user, so aiming for 60FPS was the teams benchmark to achieve a comfortable experience for the user. To achieve this we used optimised shaders, geometry, limited objects and other code optimisations. Performance testing on the Mobile & Desktop application also proved invaluable to enable realtime data to be passed as quickly and efficiently as possible between the different platforms to avoid user frustrations and confusion.
With such a quick turn around project, there wasn’t much time to do user interaction testing. With the HoloLens being a new 3D paradigm, it meant there was a lot of unknowns as to how a first time user would respond to certain interaction inputs, animations, 3d buttons, transitions, noises, FX and voice commands. So very quick rapid prototyping and 3D practical experiments where needed to test, using cardboard cut outs and boxes placed in areas helped understand how the user could best interact alongside physical objects and UI elements. The ‘air tap’ interaction of the Hololens can prove difficult for some people, so initial good people management training was important for first time users to understand this new paradigm. For all first time users of the HoloLens headset it was advised an initial user orientation training session was undertaken to allow the users time to understand the HoloLens interactions and technology.
Although the HoloLens 1 was a cutting edge piece of hardware, it is only a prototype!! one of the biggest constraints is the field of view (FOV) as this is quite restricted in the headset. The challenging problem was to keep the users gaze and attention focused on the task in hand. Using look at 360 floating cursor arrows for viewing points of interest was invaluable. For positioning users for optimised viewing angles during training, animating footprints, specific targets on the ground and POI also proved useful to keep the user engaged with a limited FOV.
Overuse of sound effect and voice overs proved irritating after several uses, so in future to be optimal with voice over instructions and voice cues, as this damaged the UX in some areas of the app, and we added an option to turn of voice instructions. Also being very subtle with sound FX for button hover and selection states as its important to let the user know they have input certain commands, but over use can destroy the user experience of the training application.
Voice commands where added to all the navigation buttons in the app, they popped up when the user button gazed for more than 0.5 seconds. The issues we had was the working area of the Flexwrap in the factory was very noisy and loud and the HoloLens mic struggled to register user voice commands. The HoloLens 2 however has directional microphones that pick up sounds from the user with any loud ambient noise around, apparently!!! (This needs to be tested as I’m still convinced VCs would work in a loud factory environment that the Flexwrap was in!!!) More testing in the factory environment would have been useful, to not waste valuable development time, on certain features.
User testing was so important throughout the development life cycle and actually completing the practical swabbing tasks proved to be invaluable that required the end to end process. This allowed us to tweak and modify certain task procedures in order to fit with the technology, in the most user efficient way.