When we set out to build Sherlock, our goal was to help SaaS businesses of all sizes operationalize their product engagement data. After running many SaaS businesses ourselves, we knew how essential this data was to all our customer-facing teams — yet how hard it was to for them to access and use it. We knew that, at the very least:
Sales needs to know which trials are actually using the product and when users become activated;
Customer success needs to know when key accounts are in danger — or ready to expand;
Product needs to know if key features are being adopted;
Marketing needs to know which customers were most successful so they can go get more
We created Sherlock to help package this data in a way that all of these teams could easily benefit.
But we also realized that to make this data truly actionable, teams needed to have it in the places they lived on a regular basis. This is why we opened up connections to bring this data to Segment, Intercom, Salesforce, and more.
But still, that wasn’t enough. Not only do these teams need the data and need it in their tools of choice – they need it fast. This is the job of our latest release, Sherlock Alerts. With Alerts, teams can ship their essential product engagement data from to Sherlock when they need it, where they want it — in Slack.
With this release, Sherlock has taken another big step in helping SaaS teams operationalize product engagement.
What Are Sherlock Alerts?
Quite simply, they are the easiest way to keep your team updated on how your users and accounts are engaging with your product. With Alerts, you can design notifications for any engagement situation. If there’s something you want to know about how your users are engaging (or not engaging) — there’s an Alert for that.
So What Can I Be Alerted About?
Good question! Here’s just a few things:
There’s been a change in a user or account’s Engagement Score
There’s nothing more fundamental to Sherlock than the Product Engagement Score. It’s a one-number overview of how engaged your accounts are with your product. And now you can get alerted whenever there’s an increase or decrease in an account’s score. Stay abreast of churn threats or — even better — expansion opportunities!
Anew account is created or new user is added to an existing account
Ahhh…is there anything more exciting than seeing new users being added to your product. One of the purest shots of dopamine.
But even more important than the self-satisfaction these alerts bring, is how they can keep your sales and CS team on top of these new entrants. Knowing when new accounts are created can help your sales team get a jump on the competition. Knowing when new users are added to existing accounts can insure that your CS team is delivering a spectacular initial experience to these important users.
A trial lead has become activated – PQL alerts
Ok, let’s get real. The vast majority of today’s SaaS offerings have some kind of free trial or freemium offering. It very difficult to compete without it. Try-before-you-buy has become the norm in today’s software universe. This means your sales team is constantly selling to prospects who have already started to use your product.
This means your salespeople need to know how far an account has progressed during a trial period. They need know the activation rates for every trial account. And they need to know once an account passes a certain threshold of Activation because this is an essential signal for conversion.
This alert is foundational for any effective PQL process. A bit of a game changer here.
Someone’s entered a segment
User and Account segmentation is one of the most important features in Sherlock because they allow you to organize and track the important slices of your customer base. In many cases, knowing when any user or account enters any of these segments is important. With Sherlock Alerts, you can.
Bonus! Create Your Own Alert
At the end of the day, your product is unique. Instead of relying on a “one-size-fits-all” approach to alerts, we wanted to ensure maximum flexibility in this feature. We wanted our users to be able to create Alerts based on any engagement scenario – and that’s what we’ve done with our custom alerts.
Don’t just measure Product Engagement — operationalize it
Our goal at Sherlock is to help every SaaS business (that’s you!) improve how they execute and compete. Knowing how users and accounts are using your product is essential information for high-performing sales, CS, and product teams. Many of their day-to-day decisions are based on this information. If they are spending their time fishing for it, that is time they are not selling, supporting, or building.
Getting this data into the hands of your people in a way they can use it, in the place they want it, at the time they need it is what Sherlock is all about. This latest release is a huge step in this direction.
If your SaaS product is like many with a free trial offering, then odds are, your customer journey probably looks something like this:
Arguably the most important phase in this journey (and the one that causes us all the most agita) is the “Activation” phase. You know this phase. It’s the one that keeps you up a night. It’s the one that carries both euphoric satisfaction (Yes…a new customer!) and crushing disappointment (Why aren’t they using the product? Why don’t they love us?!) During this phase, your goal is to ensure that the accounts testing your product get to a point of “first value.” The fact is, without getting to this point, there is no way for them to assess whether or not your product will be valuable to their business.
If there is no “first value”, there is no value, period. There is no way to convert a trial user to a paid user unless they first become “activated” within your product. This is why to the ability to track this phase of your customer journey is oh-so important—it’s essential for your entire operation to be able to tell both how many and which accounts reach this point.
Defining activated accounts
Simply put, an “activated account” is an account that has performed a series of actions which get them to the point of first value. This is different for every product. For a project management application, the point of first value might be:
• Account created • Invited 2+ team members • Created project • Uploaded 2+ files • Created 3+ calendar events • Created 1+ tasks • Completed 1+ tasks
For this product, accounts that do all of these things would be considered activated during their trial. It is clear that these accounts have used the product’s core functionality and should, at this point, be able to properly assess its value.
The specific events that determine an account’s activation are unique to every product. Activation for your product is going to be very different than activation for another product, even if they are very similar in nature.
As you can see, defining and tracking account activation is all about product engagement. Activation is defined by specific actions and engagement with your product. Given that Sherlock is all about product engagement, this is right in our wheelhouse.
Tracking activation in Sherlock
With Sherlock you can now very easily track the activation progress for your users and accounts. By setting up Activation tracking in Sherlock, you will be able to identify those accounts which have met your criteria and acheived “Activation”, those which haven’t, and every account in-between.
Setting up your Sherlock account to track engagement is actually very easy. Here’s how it works:
Step 1: Define Activation with activation criteria
Simply click on the “Account activation” link on your Account overview page and you can set the criteria for your activation. You can define this activation using specific events that an account should take, a series of account traits, or a combination of both.
Step 2: …….
Oh, right. There is no step 2. You’re done.
At this point, Sherlock will take your activation criteria and immediately calculate the Activation Rate for all your accounts. What is an Activation Rate, you ask? It’s elementary:
Once you set up your Sherlock account to track activation, you will see a new metric everywhere you navigate in Sherlock as well as in your various Destinations. We call it Activation Rate.
The Activation Rate describes an account’s progress toward Activation. It is always displayed as a percentage. For example, if your activation criteria has 4 steps and an account has met 3 of them, then its Activation Rate will be 75%. An account with an Activation Rate of 100% is considered “Activated”.
This is a metric with which you will become very familiar once you set it up in Sherlock. It will help define many of your internal processes – from sales to CS to product development.
Where to find Activation Rate in Sherlock (and beyond)
Activation Rate is a key metric that you can find in various places in Sherlock.
First of all, you will find it on every account detail page:
As well as on the Account overview page:
You will also have default segments for Activated accounts that will allow you to see those accounts with Activation rates = 100%. At the same time, you will have the ability to create your own segments based on Activation Rate.
Finally, you will also be able to access Activation Rates in any of the tools to which you are sending Sherlock data. Here are three important places that you can connect to from Segment:
SLACK By connecting Sherlock to your team’s Slack account, you will receive a daily update letting you know if any new accounts have entered any segments you’ve created, including your new Activated Accounts segment. This will alert your team to any newly-activated accounts.
INTERCOM If you do most of your account management work in Intercom, then syncing your Sherlock account with Intercom is a great way to manage this activation phase. With this integration, you will see Sherlock account segments populate to Companies in Intercom.
Having this data readily available in Intercom will enable your CS team to always know which accounts are activated (and which aren’t) while performing their day-to-day work. You can also use this attribute to trigger messaging from Intercom, as shown below.
SALESFORCE If you have a sales team working your trial accounts, then knowing when a trial becomes activated is essential for them. They need to be able to prioritize their leads on a daily basis. Knowing whether or not an account is activated helps inform when and how to engage with that account in a sales situation. It’s essential knowledge for a scaling sales team.
By connecting Sherlock to Salesforce, you will be able to import your account segments into Salesforce account records. From there, your sales team can create reports to show which accounts are actually activated and ready for that all-important follow-up call
Tracking trial account activation is essential for all SaaS products
Thought I’d end by stating the obvious.
The reality is, accounts that are trialing your product are at the most fragile, most vulnerable point in their lifecycle. At this stage, they are like newborn infants. The world (that is, the world of your product) is huge, unknown…scary, even. They need support, they need nurturing, and they need attention. How you monitor and act during this period will determine whether your new account becomes a strong, happy, successful adult, or…well, not. By tracking the health of your accounts closely with Sherlock, you will put yourself in a much better position to see to their proper development. Happy parenting, SaaStrs!
Winter is coming, so the Sherlock team is generating its own heat by pushing features at a blistering rate! We continue to inch toward our goal of making Sherlock an absolutely indispensable part of every SaaS stack.
To that end, we have a few nice releases this week:
Intercom company integration
This was one of our most requested features and we’re excited to be able to release it. Thanks to our users for continuing to push us on this. Also, a special thanks to the Intercom team for working with us on this one. This wasn’t as straightforward as we expected it to be and we appreciate their team being flexible.
You can add these segments and scores to your Intercom ‘Company’ views by selecting them in the attribution selection box:
That will allow you to always see these scores and segments in your Company views:
Not only will this allow your team to get amazing context on each one of your accounts while working in Intercom, but will also allow you to use Sherlock scores and segments to target users in your messaging. For example:
Email users who belong to an account with an engagement score above 50
Email all users on an account who are in the “New trial, not activated” segment
Email congratulations email to all users on a newly “Paid account” segment
Show pop-up to all users on an account that hasn’t entered “Used feature X” segment
You can learn more about using the Intercom integration here.
Additional account segment filters – number of users
We also had many requests for the ability to segment accounts based on the number of totalor active users on the account. So, we added that to our filter options:
With this, you can create segments of accounts that have a lot of users, or very few users. Target accounts with only one user with a message to add more. Or create a segment of accounts with more than one user – but only one active user – as a precaution. Those are risky accounts!
See all-time activity for users and accounts
Sometimes, it’s very important to see a user or account’s activity over their entire lifetime – but very often hard to find. Sherlock has now made that very easy.
On any user or account detail page, you can now toggle the date of the Activity Breakdown to ‘Ever’ – allowing you to see what events have been triggered by that user/account over all-time (at least since you have been using Sherlock). No longer do you have to wonder – “Did this account ever do XYZ?” Just toggle and discover!
Creating your engagement score is the first step in your Sherlock journey because everything in Sherlock is dependent on the creation of a scoring model.
The beauty of Sherlock is you’re in control (it’s your product after all – you know it best!), but what is the best way to go about creating this model in order to ensure the best results?
This article will outline a few tips and best practices based on some frequently asked questions.
How many events should I have in my engagement scoring model?
You are likely tracking dozens of events in your product, but generally speaking, there are probably only 10-20 that really matter to assessing engagement. We have seen successful scoring models with as little as 5-7 events and some with as many as 20+. That said, 10-20 is a good rule of thumb.
Do be aware that too few events is problematic for creating a good scoring model. If you only use 1, 2, or 3 events, there will likely not be enough diversity in the usage to create a wide enough range to create a meaningful ranking of users/accounts.
How should I weigh each event?
A properly configured engagement score should expose and elevate your “power users.” This means your “power user” features should be given the highest weight (7-10) in your model. These are the features that are not very frequently used, but used by your most advanced and experienced users.
Your most common events – those triggered by every (or the majority) of your users – such as login – should be given the lowest weight (1-3) – or not weighted at all.
Features that fit in the middle – what you might consider your “core” features – should be given medium weights (4-6).
Is there any way I can tell if there I’ve missed any important events that should be added to my scoring model?
One of the most unique features of Sherlock’s Activity tab is the ability to identify the events that are driving the most engagement. A few days after creating your first score, you should check this tab – toggle to the “unscored events” and look for engaging events that you might want to add to your scoring profile.
How rigid is this score? Do I only have one shot to make it correctly or can I edit it later without affecting the results?
It is very easy to update and iterate on your Sherlock score. In fact, during your first few days with Sherlock, you will likely be iterating on your score quite a bit. When you change or update your scores in Sherlock, Sherlock quickly refreshes the calculations that drive all your reporting data. You don’t have to worry about your changes destroying your data in any way.
You shouldn’t worry too much about creating multiple scoring profiles in your first few days with Sherlock, but it is something to keep in mind. Multiple scoring profiles come in very handy when you have different groups of users that use your product differently.
Remember, no one knows your product as well as you do. Because of this, creating your scoring model should be very intuitive. In general, you should trust your instincts here.
Most SaaS business are account-based. That is to say, most SaaS products are built to be used by a collection of users, teams or even entire organizations. But tracking engagement data at the account level is difficult.
At Sherlock it’s our goal to make your product engagement data as accessible and useful as possible. We know that user-level engagement insights are key, but that for 99% of you account-level insights are just as – if not more in some cases – important.
Sherlock has always offered account based engagement scoring, but over the past month we’ve put a lot of time in upgrading our Account detail page.
The new and improved Account detail page
From the ‘Account’ tab (or via search), click into an individual account and see:
Total users vs active users
Account engagement score over time (both average and total score)
Insight into account engagement and which users are driving that engagement is a key component to your CS, Sales and Marketing efforts. To learn more about this new view, check out our help doc here.
If you’re already a Sherlock user then make sure you check it out and if you’re not a user – what are you waiting for!? Start your free trial or book a demo!
As always we will continue to work hard to make Sherlock the best product possible. Let us know if you have any questions or feedback!
Measuring product engagement is, in some ways, a nuanced game. Especially if you want to do it right — if you want to do it at a level that makes it as helpful and actionable as possible.
One of the nuances of the engagement scoring game is the fact that different groups of users (or accounts) engage with your product in different ways. In fact, in many cases, this is by design! Different groups of users are supposed to engage with your product in different ways.
Most likely NOT a reflection of your user base.
New users do things during their onboarding/activation phase that are different than what “mature” users do during ongoing engagement;
Trial users may use a product differently than paid users;
Different user types (ie — teachers vs students; buyers vs sellers; players vs coaches; etc) may use completely different feature sets;
Users with different access-levels (ie — Admin, Manager, Team Member, View-only) have very different usage profiles;
And many other examples. I’m sure you can think of several unique usage models for your product as you read this.
These different usage models are incredibly important in the context of truly understanding engagement with your product. In these cases, “engagement” needs to be defined differently for each unique usage model. Each usage model needs its own unique scoring profile.
Engagement for a new user should be defined differently than engagement for a mature user.
Engagement for teachers who are creating lesson plans in a product should be defined differently than for students who are using the plans.
Engagement for Admin users who manage account configuration tasks should be defined differently than for Team Members who should be using the product to manage daily activities.
And this is why, in Sherlock, you can target different groups with unique, individual engagement scoring profiles.
Creating multiple scoring profiles in Sherlock
In Sherlock, you create an engagement scoring profile by defining your most important product events and giving them a weight based on how important each event is to your product.
Creating an engagement score in Sherlock
But instead of creating only a single scoring profile that applies to all your users, you can create different scoring models for different sets of users.
These users can be defined by user traits (user type = “admin”) or account traits (account status = “trial”) or other parameters (signed up < 30 days ago, etc).
Define the unique group and target a filtered list of users
Then you can define a unique scoring profile for each of these groups. You can select different events that define engagement for that group, or weigh the same events differently, or both.
The point is that this functionality will allow you to truly hone in and understand engagement in a more nuanced way than you have been able to before.
Toggle between your various scoring profiles
Once you have multiple scoring profiles in Sherlock, you can toggle between them and get a complete picture of engagement across different slices of your user base.
You can see:
the top users in each profile;
the top accounts for each profile;
which events are driving engagement in that profile;
Part of our goal at Sherlock has always been to make engagement data “actionable” — make it something you can use to improve your operations. And that extends to these multiple scoring profiles.
When you create multiple scoring models, you can select which profiles you want to send to your various Destinations (Intercom, Salesforce, Slack, etc).
There may be specific scores that you want in specific destinations and not others. For example, you will likely want to push only scores from trial accounts to your CRM, but all your scores to your support tool, etc.
The complete picture
You know as well as anyone that not all users engage — or are even meant to engage — with your product in the same way. And therefore, in order to truly assess your product’s engagement, you need to be able to create engagement scoring models specific to these unique usage patterns.
Only then can you get a complete picture of engagement with your product and drive the right actions to push that engagement forward.
Privacy & Cookies Policy
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.