UPST changed their AI/ML models again?

I have been keeping track of the number of daily Trustpilot reviews on Upstart. Borrowers who complete a loan transaction with Upstart are all invited to write a review on Trustpilot.

According to Trustpilot: “Trustpilot displays reviews chronologically, with the most recent review at the top. We don’t monitor reviews as they come in — they’re not manually moderated or censored before being published. Once posted, reviews are instantly visible to everyone.

Below is the averaged number of Trustpilot reviews per day for each month. This has seen a general steady climb upward over the past year.

June 11.23333333
May 11.5483871
April 11.53333333
March 9.741935484
February 10.5
January 8.322580645
December 7.903225806
November 3.933333333
October 3.677419355
September 2.033333333
August 3.516129032

It is on average, 3-4 reviews a day from August to Nov 2020, then a jump to about 8-10/day from Dec to March.
Over the last quarter it has been about 11 reviews per day from April to June.

However, starting on July 1 2021, the number of reviews per day suddenly exploded to crazy high numbers:

14-Jul 98
13-Jul 101
12-Jul 84
11-Jul 49
10-Jul 64
9-Jul 79
8-Jul 86
7-Jul 87
6-Jul 75
5-Jul 51
4-Jul 30
3-Jul 38
2-Jul 67
1-Jul 58

I don’t know what’s going on here. For the past 2 weeks this July, the daily number of reviews is averaging 69! Over 6 times the average daily reviews in just the past month of June.

Does this mean anything? Did Upstart somehow encourage a higher proportion of users to write reviews? Or did Upstart improve their AI/ML underwriting and ad targeting to drive conversion rates even higher?

I tried to correlate this data with Google Trends, but the Google search popularity for the terms Upstart, Upstart login, Upstart loan doesn’t appear to have spiked 6 times since July 1.
https://trends.google.com/trends/explore?date=today%205-y&am…
https://trends.google.com/trends/explore?date=today%205-y&am…
https://trends.google.com/trends/explore?date=today%205-y&am…

While Alexrank’s traffic analysis on Upstart.com has also shown a very steady climb upward the past 90 days (23.88% jump from #22385 rank to #17039 rank), there’s also not a sudden spike as seen on Trustpilot.
https://www.alexa.com/siteinfo/upstart.com#section_traffic

Any thoughts would be appreciated.

59 Likes

The bottom line is, you have no way to tell without more data.
It could be:
– More loan volume from Upstart (Increased Volume)
– A new layout/positioning of the trustpilot invitation that increase clicks (Improved Click-thru)
– People being massively excited by their experience and just wanting to write about it. (Improved Product)

4 Likes

Could it maybe just be that upstart implemented some sort of incentive for posting a review?

First post on this board, been reading every post for close to a year which was when I first started investing. It’s no exaggeration to say that this board has definitely changed my life!

8 Likes

Occam’s Razor - “more loan volume”

They are signing up and onboard more banks on their platform.

3 Likes

Thank you JonWayne235 for your deep research on Upstart.

I looked through some of the reviews on Trustpilot and I think the recent surge in reviews is a result of a combination of both, (1) an increased loan volume, and (2) either an incentive to post a review or an improved click-through for the Trustpilot review invitation.

  1. Higher loan volume:

In the interview with Upstart co-founder Paul Gu, Paul mentions that the thing that surprised him most about the consumer-lending space is how prevalent and successful mail-in offers are even in the 21st century. Looks like Upstart has started doing the same as well. Upstart is sending pre-screened offers to people by mail, and it looks like it is working, and this may be a reason behind higher loan volume. Below are some examples of reviews on Trustpilot which mention that they received such pre-screened offers from Upstart and then applied for a loan.

“I received a letter in the mail saying I was prequalified for a loan from upstart. I was skeptical but logged on and filled out the info and was approved in minutes! It’s not a scam. This really works!”

“I received a letter in the mail from upstart, I applied for a small loan and was approved within just a few minutes. Pretty impressed. Thank you so much!”

“Received a “prescreened offer” from Upstart. Unlike the usual “prescreened” offers that never pan out, Upstart did just what they said they would do. I received a GREAT loan from an AWESOME lender in minutes. Money in my bank the NEXT DAY! Thanks Upstart!”

  1. An incentive to post a review or an improved click-through for the Trustpilot invitation link

There is no way to distinguish between them and know which one is the reason unless someone has recently applied for a loan through Upstart and would be willing to share their experience. After looking through some of the reviews on Trustpilot and then checking the reviewer’s profiles, I found that almost all of the reviews are the first and the only review posted by that reviewer. However, I found one review saying that it is the second time they applied for a loan from Upstart, but it is their first review on Trustpilot (see below). Although one review does not necessarily mean anything, it is possible that the borrower did not write a review after the first time but did so after the second time because of an incentive, or better placement of the invitation.

“This is the second time I have applied for a loan with upstart the process so easy the person I spoke to on the phone could not have been any nicer helping me through the process. I one hundred percent recommend upstart for your loan don’t waste your time with any other companies upstart is the best.”

I appreciate any thoughts from the much more knowledgeable and experienced investors on this board.

Supratim
Long UPST 10%

34 Likes

Agreeing with Supra here.

1. Higher loan volume:
I know 2 people that have received a pre-screened letter - though neither really wanted a loan. I am thinking that UPST is using AI models to now pre-screen and identity good candidates in both terms of low risk and higher likelihood of following up on the email. Using AI to target potential customers – nice extension of user of their models.

2. An incentive to post a review or an improved click-through for the Trustpilot invitation link

I think both

A. All of the 4 and 5 star reviews (or 99.9% of them) are identified as “Invited” – meaning “This company sent a review invitation outside of Trustpilot’s systems. Reviews collected this way are automatically labeled invited.

B. There is clearly a large uptick on the number of Invited reviews. If you go back a number of months you see a higher percentage of 1/2/3 star reviews that do not have the “Invited” identifier. And the nature of the beast is that invited review will be almost all positive. The company controls who gets the invite and probably not inviting people who were denied.

C. It is clear to me from the short reviews that Upstart is inviting users to review at multiple stages – some comment state they haven’t received the money yet, others indicate it just arrived, and others report the funds a week ago”. So I believe they have upped the contact points where a review is requested - requesting the review once the loan is approved in the app and then sending follow-up emails with further invites if the in-app was not utilized. I get these requests from other companies I have done business with and to be honest it gets annoying – but it works.

One reason I like TrustPilot is because they do identify the Invited reviews. But it is also why I discount these reviews – they are usually short and meaningless with no real details. And companies use them to drown out the negative reviews – which often have more detail and are more meaningful.

Upstart probably had one of their AI/ML experts spend 2 days writing models to identify the best candidates for Inviting to review. :wink:

BornGiantsFan

Long UPST

22 Likes

Appreciate everyone’s thoughts.
I think you are right on the increased automated invitations/higher response rates as playing a large role. For the month of June, 9.2% of Trustpilot reviews were uninvited. For July so far, 0.93% were uninvited reviews.

I poked around the Trustpilot site some more and found they actually give a transparent graphic breakdown of reviews over the past 12 months:
https://www.trustpilot.com/review/www.upstart.com/transparen…

Regardless, I believe the Google Trends search terms for Upstart and Upstart’s Alexarank traffic data give a (speculative) clue that Upstart may not have seen a slowdown in growth despite macroeconomic headwinds.

We know New York Fed data and big bank earnings have so far shown reductions in consumer lending/credit card balances in 2021 (although there are increases in mortgage/auto categories).

We can see CreditKarma’s on Alexrank has fallen from #528 to #659 over the last 90 days which correlates https://www.alexa.com/siteinfo/creditkarma.com#section_traff…

Other aggregators Credible.com, Lendingtree.com have a dip in traffic as well.
And if you look up other fintech lenders, they have similar coinciding website traffic drops.

That being said, this is all pure speculation.
We’ll know when earnings/guidance comes next month.

20 Likes

“I know 2 people that have received a pre-screened letter - though neither really wanted a loan.”

This is interesting. It suggests to me that the low-risk clusters which UPST identifies in their datamining are also being used to develop targeted marketing campaigns based on the demographics of each particular cluster. This is a logical and expectable application of their technology.

I think I’ll put more money into the company; I like the way they’re operating.

1 Like

Hi all – First time poster (please forgive text effects as I’m a novice at this)

First off, thanks jonwayne235 for this valuable metric – Trustpilot reviews – great way anticipate growth in some manner

Increased volume may be due to Auto Loans. From the May 11 Q1/2021 Earnings Transcript, I found this move by Upstart pretty ingenious …

Dave Girouard - Chief Executive Officer … (about 3 pp down in report)
" I’m also happy to report that we closed our acquisition of Prodigy a leader in automotive commerce software. Prodigy is like Shopify for car dealerships helping to create the modern multi-channel car buying experience that dealerships need and consumers rightfully expect in 2021. In addition to modernizing the car buying experience, Prodigy will allow us to bring Upstart’s AI enabled auto loans to dealerships across the country, where the vast majority of loans are transacted. By potential distractions from the merger, the small but mighty Prodigy team increased our dealership footprint by 45% in the first quarter. Even at this early stage, almost $800 million in vehicles were sold through Prodigy in Q1 2021.
During the quarter, we became a certified digital retail provider for Subaru of America retailers. As I said last quarter, we believe Prodigy will enable Upstart to tap into one of the world’s largest buy now pay later market opportunities. To realize this potential we’re significantly increasing investment in Prodigy’s technology and go-to-market teams. The first quarter wasn’t just a win from a financial perspective. We also made significant strides in our ongoing fair lending efforts…"

Ingenious way to agressively pursue market share / TAM. My eyes popped when he talked of Prodigy being the Shopify for Car Dealerships, implying end-to-end customer service at the car dealership, including car loans with instantaneous approval per Upstart platform. Really like how Upstart doesn’t let laggards like banks (notoriously slow moving) to stand in its way, and forges ahead to grab market share.

This with the NXTsoft partnership to facilitate / API to banks and other financial institutions…

— and —

In June, NAFCU Services Announces Upstart as a Preferred Partner for their AI Lending Platform for Credit Unions

Looks like Upstart is doing all the right things.

And with fist mover initiative into AI loans, with “nobody doing this right now” as per Bert Hochfeld and Dave Girouard CEO, very much looking forward to the 8/10 earnings release / call.

Comments, critiques are very welcome

Thanks
John
Long UPST 14%

PS – thanks to Saul and team for this top notch board and monitoring to keep us OT.

34 Likes

Welcome, John! (says the also-new person)

To add to your comments, it looks like at least one credit union has gone with the recommendation. I got this notice today of adoption by a credit union in Ohio: https://ir.upstart.com/news-releases/news-release-details/te….

Anne
long UPST (biggest position by cost basis)

5 Likes

“Does this mean anything? Did Upstart somehow encourage a higher proportion of users to write reviews? Or did Upstart improve their AI/ML underwriting and ad targeting to drive conversion rates even higher?”

Now we know the answer, from the earnings call:

“I can just address the initial assumption, Ramsey, which is we did have good continuing progress in terms of our conversion rates on our funnel that we’re sort of pushing up against 25% this quarter”

So their conversion rates were:
Q3 2020 - 14%
Q4 2020 - 17,4%
Q1 2021 - 22%
Q2 2021 - 24%
Q3 2021 - 25% (sort of guidance)

It looks like there is not much space to improve the conversion rate - obviously they can’t give loans to everyone - so the answer could be that they’ve increased the number of review invitations or marketing efforts.

According to Similarweb (https://www.similarweb.com/website/upstart.com/#overview), which gives a pretty accurate picture as I know, their website traffic increased by 12%, from 1,95M visits to 2.2M visits. That couldn’t increase the number of reviews so much.

So the answer is that they just increased the number of review invitations, that’s it. They had 286864 loans in Q2, that’s about 95K per month, and it looks like about 2% of customers are now leaving reviews - that’s cool.

Regarding their marketing - I made some research.

In addition to mentioned above direct mail activities, they started advertising with AdWords (data from ahrefs.com) in July and August, but it looks like they are just experimenting and are not investing much (orange bar is the % of days when their ads were visible):
https://imgur.com/a/M6iuvhC

They use live reviews from trustpilot on the landing pages:
https://imgur.com/a/WHGCvLi (https://www.upstart.com/i/pl1-brand)

Collecting more reviews is good for improving their landing page conversion rate.

So, according to my research, they do have some sequential growth in traffic (12% per month is good - that’s 40% per quarter, but it is just math, we’ll see), but we shouldn’t expect that the increased number of reviews will lead to similar increase in the revenue.

9 Likes