UC Santa Cruz

Interface Evaluation of Cleo

Timeline

Sep 2019 - Dec 2019

Role

Designer

Team

Neel Saswade

Tools & Skills

Usability testing

Overview

This project evaluates the interface of the popular fintech application, Cleo. Usability testing was performed with real users, over the course of 8 weeks. First, I interviewed experienced Cleo users to understand how they use Cleo and extract 3 main tasks. Next, I conducted usability tests with participants who have never used Cleo before, and recorded qualitative and quantitave data. I suggested improvements backed with research data and foundational interaction design principles. All findings were typed up into a research paper along with a slide deck to share with the general stakeholders of Cleo

Cleo

At 2.5 millions users, Cleo (2016) sets itself apart from most financial tech (fintech) applications by using artificial intelligence to help users manage their money. Cleo’s goal is to be the main hub for people (mostly millennials) to make financial decisions. To use the app, users connect their bank accounts, set budgets, add their bills, and interact with Cleo.

*(left) regular central interface, (right) central interface after pressing “Roast Me”

The central interface is a messaging platform, where one chats with the bot to learn more about his/her financial situation. Common phrases someone can ask Cleo are, “Can I afford another coffee today?” or “I’d like to change my spending budget.” Cleo’s party piece is the bot’s entertainingly witty comments where it takes no effort in shaming or roasting the user when they have bad spending habits. This intimate relationship between computer and human is what intrigued me to find out if regular people can productively interact with a chatbot in a field where chatbots are rarely used.

Apart from the chatbot interface, Cleo has two other screens that the user can swipe to. The left side displays the user’s spendings and savings. Transactions are easily traceable through this page. The right side has a discover page where users can learn about certain features (i.e., create a wallet, Cleo+ subscription, etc.) along with the Roast me and Hype me buttons.

Cleo seamlessly integrates the core chatbot feature into their financial application. The app is fun to use and handles people’s finances in an organized manner. Because Cleo is pioneering the use of artificial intelligence in the fintech realm, I will be examining the user evaluations of this application and proposing any future design improvements.

How do people use Cleo?

Experienced user interviews help us identify the main tasks people use Cleo for. Two experienced users were interviewed by phone call based on a semi-structured interview guide.

 

Miguel

  • Uses budget feature when Cleo notifies him of poor spending habits

  • Creating and monitoring bills gives him a good financial outlook on the month

  • Loved chatbot feedback and enjoyed interacting with it

Notable Quotes

  • “I didn’t realize how much I was spending at the local department store”

  • “I had to change a bill after getting new insurance that comes on the first of every month and it was really simple to change that.”
    “It’s kind of like having your own personal assistant at the palm of your hand.”

 

Sarah

  • Manually checks spending through app

  • Mostly uses the budgeting feature

  • Likes the data visualization of her spending

  • Creates and monitors bills helps her to keep track of subscriptions

  • Absolutely loves the chatbot and their impersonal relationship

Notable Quotes

  • “I like how I can see it month by month in a little chart too and see what was happening that time.”

  • “I absolutely love interacting with the chatbot. I’d give that 4.5/5 stars. I was really happy with having those different creative ways of putting how money is discussed.”

  • “I love the feeling I get when I press hype me. It makes me feel like I can do this adulting thing.”

Identifying the Main Tasks

 

1. Setting a Budget

2. Adding/Removing Bills

3. Interacting with the Chatbot

User Task Evaluation

Novice user task evaluations were performed on five novice users who had never used Cleo prior to the study. Evaluations were in person, semi-structured interviews with video/sound recorded for analytical purposes

New User Interview Process

Each interview began with a briefing of the participant on the study followed by questions on demographics and how they track their financials. Participants were then handed a set of instructions and told to read them out loud. Once the webcam began recording, participants were asked to carry out the instructions using Cleo. The number of errors, clicks, and time of each task was recorded as the participants progressed through the instructions. After the participant completes all three tasks, the webcam recorder was stopped and the participants were asked questions based on how they felt while engaging with the app.

The Data

Quantitative

Qualitative

Task 1: Setting a Budget

  • “Wow that was easy to find” - Participant D

  • Visible frustration when could not find any buttons to set budget - Participant C

  • Used +/- buttons but they only added/subtracted by 10 & did not reach desired budget - All Participants

  • Unsure of what to do, left budget as is - Participant F

Task 2: Adding/Removing Bills

  • Error when attempting to add bill because “add” not present - All Participantsall participants performed errors when trying to add a bill because a visible “Add” was nowhere to be seen

  • “This could have been a lot more straightforward” - Participant D

  • Clear confusion with how to edit bills after selection - All Participants

  • Deselecting/Removing a bill was easy - All Particpants

Task 3: Interacting with Chatbot

  • Finished task quickly and stated as easiest task - Participants D, E, G

  • Visible frustration and could not complete task - Participant C, F

Recommendations

What Cleo Does Well

  • Clear affordances to navigate throughout the general interface

  • Clean and aesthetic design

  • A well-designed and engaging chatbot

What Cleo can Improve

Setting a Budget

Problem: Participants had trouble setting the correct budget with provided +/- buttons

Solution: Implement a thin box around the budget feature to indiciate to users that they can manually edit the budget if the +/- buttons do not reach their desired budget

Adding/Remove Bills

The Problem:

Solution:

Adding/Remove Bills

Problem: Editing bill after selection, adding a bill after in the bills menu, bill selection were all confusing due to poor wording and saliency.

Solution: Replacing “confirm” button with “next”, to let users know they can further edit their bills. Replacing “Select” button with “Add.” Redesign “Remove Bill” button to be larger and a more physical button.

Interacting with Chatbot

Problem: Scope of chatbot features unclear to some participants. Help button guides to users to general FAQ

Solution: Typing “help” into the chatbot should display and demonstrate the complete feature set of Cleo’s chatbot

Retrospective

Being more research oriented, I brought upon my experiences working in a research lab and modified the research method to function in a usability study. During usability testing, I was amazed with how certain features caught different novice users off guard and my mind immediately conceptualized ways to mitigate these errors. Moving onward, this project has integrated a thorough research component in my design process. To take this project further, I would create high-fidelity mockups of the full Cleo platform redesign