This blogpost is being discussed on hackernews where some employees and apparently the CEO of Cloudflare chime in. See discussion: https://news.ycombinator.com/item?id=29465729
OK…
So, I’m the tech lead for Cloudflare Workers. In complete honestly, I did not even know we ran some sort of comparison benchmark with Compute@Edge until Fastly complained about it, nor did I know about our ToS clause until Fastly complained about it. I honestly don’t know anything about either beyond what’s publicly visible.
But as long as we’re already mud slinging, I’d like to take the opportunity to get a little something off my chest. Fastly has been trumpeting for years that Compute@Edge has 35 microsecond cold starts, or whatever, and repeatedly posting blog posts comparing that against 5 milliseconds for Workers, and implying that they are 150x faster. If you look at the details, it turns out that 35 microsecond time is actually how long they take to start a new request, given that the application is already loaded in memory. A hot start, not a cold start. Whereas Workers’ 5ms includes time to load the application from disk (which is the biggest contributor to total time). Our hot start time is also a few microseconds, but that doesn’t seem like an interesting number?
We never called this out, it didn’t seem worth arguing over. But excuse me if I’m not impressed by claims of false comparisons…
On a serious note, I’ve been saying for decades that benchmarks are almost always meaningless, because different technologies will have different strengths and weaknesses, so you usually can’t tell anything about how your use case will perform unless you actually test that use case. So, I would encourage everyone to run your own test and don’t just go on other people’s numbers. It’s great that Fastly has opened up C@E for self-service testing so that people can actually try it out.
Apparently they got called out that there is a statement in ToS that doesn’t allow benchmarking CF’s services and these were the responses:
> I think you’re misinterpreting the clause. But not being a lawyer, I don’t really want to get into that discussion.
The clause says “Unless otherwise expressly permitted in writing by Cloudflare, you will not and you have no right to: […] (f) perform or publish any benchmark tests or analyses relating to the Services without Cloudflare’s written consent;”[1]
IANAL, but this seems to very unambiguously prohibit benchmarking Cloudflare’s services unless you have written permission. I know you don’t want to get into an argument on HN, but could you like… bring it up to someone inside of CloudFlare who would be capable of changing it? You can point to this thread about how this clause is generating negative publicity.
[1] https://www.cloudflare.com/terms/ section 2.2
reply
eastdakota 6 hours ago | root | parent | next [–]
I am a lawyer and also the CEO of Cloudflare. I have no idea why that clause is in our ToS. It was a surprise to me when it was pointed out recently. Not sure when or why it got included. Best guess is it was when some “stress tester” services decided to “benchmark” us by performing DDoS attacks and we thought we needed another justification to shut them off. Has been a loooong time since we worried about such things. Regardless, we decided weeks ago we’re removing the clause during the next ToS refresh. That’s scheduled for the coming weeks. And, in the meantime, have no issue with anyone benchmarking our performance. And seems we should do a more thorough and unimpeachable set of comparisons ourselves. Stay tuned.
reply
ldoughty 5 hours ago | root | parent | next [–]
I appreciate this reply, and hope the initial engineer gets no flak for his personal opinions attempting to defend your company. It’s nice to see a tech company with employees defending it and leadership making such public statements as this.
reply
eastdakota 4 hours ago | root | parent | next [–]
I upvoted Kenton’s post. He’s the reason Workers exists. Surprised anyone worried about him commenting. I’d only be worried if I were a competitor who pissed him off by publishing BS stats. I’d imagine there’ll be an incredibly thorough and totally unimpeachable benchmarking study that comes out of this. And, anywhere we’re not the fastest, we soon will be. Game on.