Google::Cloud::Storage is about twice slower than gsutil cp for downloading files

[casperisfine]

Hi, I ran the following benchmark from a GKE container:

#!/usr/bin/env ruby

require 'benchmark'

def measure(&block)
  duration = Benchmark.realtime(&block)
  puts "took: #{duration.round(1)}s"
end

puts "--- Generate 500MB file"
puts system('dd if=/dev/urandom of=/tmp/random-500M.bin bs=1048576 count=500')

url = "gs://<my-test-bucket>/test/random-500M.bin.#{rand}"
puts "--- Upload with gsutil"
measure do
  system("gsutil cp /tmp/random-500M.bin #{url}")
end


puts "--- Download with gsutil"
measure do
  system("gsutil cp #{url} /tmp/random-500M.bin.#{rand}")
end

puts "--- Upload with GCS-ruby"

bucket = Google::Cloud::Storage.new(
  project: '<my-test-project>',
  keyfile: ENV['GOOGLE_APPLICATION_CREDENTIALS'],
  timeout: 5,
  retries: 0,
).bucket('<my-test-bucket>', skip_lookup: true)

bucket = Buildkite::FSCache.send(:bucket)

file_name = "test/random-500M.bin.#{rand}"
measure do
  puts bucket.create_file('/tmp/random-500M.bin', file_name)
end

puts "--- Download with GCS-ruby"
measure do
  puts bucket.file(file_name).download("/tmp/random-500M.bin.#{rand}")
end

In short, upload an then download a random 500MB file, with the following results:

• gsutil cp upload 7.1s
• gsutil cp download 6.1s
• G::C::S upload 4.8s
• G::C::S download 13.9s

There obviously a bit of variance between the runs, but G::C::S#download is constantly twice slower than the rest.

Also note that gsutil doesn't use crcmod here (it's printing warnings about it).

I tired looking at gsutil's source to see what it's doing differently, but it's a bit like looking for a needle in a haystack.

Any ideas?

cc @DazWorrall @wvanbergen

[blowmage]

Are you disabling the verification check when downloading? You pass verify: :none to the download method to do this.

https://googlecloudplatform.github.io/google-cloud-ruby/#/docs/google-cloud-storage/v1.9.0/google/cloud/storage/file?method=download-instance

[casperisfine]

I'll try this, thanks!

[blowmage]

Ugh, the app I use for GitHub notifications didn’t show the benchmark code in the issue description. Now that I’m reading this in a browser I see that you were not disabling the verify check.

It is possible that this verification stuff is the cause for the difference. Calculating MD5 on a 500 MB file will take some time, but we think we are doing this as efficiently as possible.

[casperisfine]

I ran the benchmark twice with verify: :none:

• run 1: gsutil -> 8.0s, gcs-ruby -> 11.6s
• run 1: gsutil -> 6.3s, gcs-ruby -> 13.0s.

So I'm afraid disabling the check only improve things marginally :/

[blowmage]

K, I'll look into this. FWIW, google-cloud-storage uses google-api-client to access the API, which uses httpclient for the HTTP calls. So it will take a bit to dive through all the dependencies to see where we can pick up performance.

[blowmage]

@casperisfine Oh! One more thing! Your benchmark includes creating the File object, which also makes an API call. Can you update it to something like the following?

  puts bucket.file(file_name, skip_lookup: true).download("/tmp/random-500M.bin.#{rand}", verify: :none)

[casperisfine]

Can you update it to something like the following?

Good point, but still no sensible change (~6s vs ~12s). But it's still nice to do one less call so I'll add this to our production system.

which uses httpclient for the HTTP calls. So it will take a bit to dive through all the dependencies to see where we can pick up performance.

Yeah my hunch says the performance difference is more likely to be at that layer. Either some non-optimal buffering, or different compression, etc

What is interesting too is that it impacts only the download performance, the upload performance is fine.

I'll do some more benchmark to see if the perf diff is linear, let me know if there is anything else I can do to help you figure this out.

Also maybe gsutil is doing something smart and the authors would know?

[casperisfine]

I did 3 more runs, this time with 100M, 500M and 1G (speadsheet)

There is a lot of variance obviously, but is does appear to be a fairly linear progression:

[quartzmo]

Hi @casperisfine, what is this line for?

bucket = Buildkite::FSCache.send(:bucket)

Can you share your entire configuration somewhere so that I can run this script in my own GKE deployment? I'm just trying to reproduce your results first, then I'll try to debug.

[DazWorrall]

[context: I work with @casperisfine]

You can ignore that line I think @quartzmo , it looks like some leftover cruft as a result of extracting this code from the project we're using it in. The result is the same as the code immediately above, it returns a Google::Cloud::Storage::Bucket object.

[quartzmo]

@DazWorrall, OK, sounds like maybe it isn't easy to share your deployment configuration.

@casperisfine I know my home workstation isn't your deployment target, but here's what I get locally:

--- Upload with gsutil
- [1 files][500.0 MiB/500.0 MiB]    8.7 MiB/s                                   
Operation completed over 1 objects/500.0 MiB.                                    
took: 65.2s
--- Download with gsutil
Copying gs://downloadtest_978004358714/test/random-500M.bin.0.5400586331450901...
\ [1 files][500.0 MiB/500.0 MiB]    6.7 MiB/s                                   
Operation completed over 1 objects/500.0 MiB.                                    
took: 75.3s
--- Upload with GCS-ruby
#<Google::Cloud::Storage::File:0x00007fd63a70d588>
took: 57.4s
--- Download with GCS-ruby
#<File:0x00007fd63a5c7368>
took: 61.9s