The screenshot on the repo only shows a comparison to the AWS CLI. Here is a comparison for uploading a single 1KB file with the AWS CLI, a minimal AWS API client and raptor:
~ $ time aws s3 cp data/1kb/file_1.bin s3://${BUCKET_NAME}/test-upload/ --quiet
real 0m0.720s
user 0m0.593s
sys 0m0.070s
$ time ./s3upload data/1kb/file_1.bin s3://${BUCKET_NAME}/test-upload/ --quiet
Uploaded data/1kb/file_1.bin
real 0m0.327s
user 0m0.136s
sys 0m0.062s
~ $ time ./raptor data/1kb/file_1.bin $RAPTOR_ENDPOINT --server-key $RAPTOR_SERVER_KEY --client-key $RAPTOR_CLIENT_KEY --silent
real 0m0.303s
user 0m0.013s
sys 0m0.014s
AWS S3 CLI or API, it's the transport and protocol overhead that are burning the extra CPU time (power). At least, that's my conclusion.
The raptor tool has another nice advantages not possible with the AWS API:
Optional Acknowledgement. The `--confirm` option allow skipping waits for ACKs, which takes the run time down to just what it takes to put the outbound packets on the network. That makes uploads very fast, and the reliability is tunable with the `--overhead` option. On networks with significant packet loss this can make urgent uploads much, much faster.
The `--rate-mbps` controls the transmission rate. Sometimes slower is better, and having this option means not needing to setup bandwidth control using `cgroup` or similar.
Interesting, I actually had to solve this problem at work. I wonder for use cases where users really need to upload that many small files, whether providing more SDKs gives more benefits than a CLI. (I see you provide a C# library, unfortunately that's out of my expertise). If your benchmark can include stats for uploads that normally take more than 15 min, that'd be much more eye-catching at least to me. Do you have benchmarks for the total round trip time as well, rom sending to full reconstruction?
Some more meta-level questions if don't mind:
1. People can tar + ztsd infinitely many files and upload that archive normally and avoid the small-file problem altogether. If somebody cannot use this approach, I guess their use case involves much more real-time reading of each file after creating it. Your pipeline involves quite a lot of moving parts. Does the tool/sdk have enough support for observability/failure recovery?
2. Pardon my dumb question. The final multi-part upload in S3 still uses the same number of HTTPs connections as if the file were sent from clients, right?
> If your benchmark can include stats for uploads that normally take more than 15 min, that'd be much more eye-catching at least to me.
I don’t have benchmarks for anything like that runtime. 100 files of 100K is the biggest. For raptor the biggest issue is picking the ‘—rate-mbps’ which controls how fast packets are sent. If defaults to a pretty low value so if you have a Gbps connection it will need to be specified to go fast.
> Do you have benchmarks for the total round trip time as well, rom sending to full reconstruction?
One use case I can talk a bit about involved transforming a large (100s of millions of records) hash database to individual S3 objects. Each was around 2kb. It took eons, even with Lambda executions sharing the work at scale.
The full end-to-end time can be observed using ‘—confirm FILE’ which will make raptor wait until the multipart upload is completed (and confirmation notification received). The baseline number is around .5s but it varies. Larger blocks with repair packets in use take longest.
> People can tar + ztsd infinitely many files and upload that archive normally and avoid the small-file problem altogether.
Sure, is that works for the use case. But sometimes small files i S3 are the desired outcome. For example, to serve as web pages or a static JSON API.
> Your pipeline involves quite a lot of moving parts.
It seems simple to me. It’s just a couple queues and a couple Lambda functions (one very simple). The FIFO queue is important and does double duty - ensuring only one completion event gets through, and keeping them in order. I didn’t add much error recovery. Resending files is often the fastest path. There are some ways to get there though. If you use the ‘—no-prefix’ flag only incomplete blocks will be processed on repeated uploads — not a solution, but something that could be built on.
> Pardon my dumb question. The final multi-part upload in S3 still uses the same number of HTTPs connections as if the file were sent from clients, right?
Yes, you have this right. Since we don’t control the S3 API all we can do is show how much better it could be if S3 had an API like this. It could save an insane amount of electricity.
The screenshot on the repo only shows a comparison to the AWS CLI. Here is a comparison for uploading a single 1KB file with the AWS CLI, a minimal AWS API client and raptor:
AWS S3 CLI or API, it's the transport and protocol overhead that are burning the extra CPU time (power). At least, that's my conclusion.Code for the s3upload tool is here: https://gist.github.com/mlhpdx/aaf99a468c06747dbea12f9639102...
The raptor tool has another nice advantages not possible with the AWS API:
Optional Acknowledgement. The `--confirm` option allow skipping waits for ACKs, which takes the run time down to just what it takes to put the outbound packets on the network. That makes uploads very fast, and the reliability is tunable with the `--overhead` option. On networks with significant packet loss this can make urgent uploads much, much faster.
The `--rate-mbps` controls the transmission rate. Sometimes slower is better, and having this option means not needing to setup bandwidth control using `cgroup` or similar.
Interesting, I actually had to solve this problem at work. I wonder for use cases where users really need to upload that many small files, whether providing more SDKs gives more benefits than a CLI. (I see you provide a C# library, unfortunately that's out of my expertise). If your benchmark can include stats for uploads that normally take more than 15 min, that'd be much more eye-catching at least to me. Do you have benchmarks for the total round trip time as well, rom sending to full reconstruction?
Some more meta-level questions if don't mind: 1. People can tar + ztsd infinitely many files and upload that archive normally and avoid the small-file problem altogether. If somebody cannot use this approach, I guess their use case involves much more real-time reading of each file after creating it. Your pipeline involves quite a lot of moving parts. Does the tool/sdk have enough support for observability/failure recovery? 2. Pardon my dumb question. The final multi-part upload in S3 still uses the same number of HTTPs connections as if the file were sent from clients, right?
Anyway, interesting read. Learned something new.
> If your benchmark can include stats for uploads that normally take more than 15 min, that'd be much more eye-catching at least to me.
I don’t have benchmarks for anything like that runtime. 100 files of 100K is the biggest. For raptor the biggest issue is picking the ‘—rate-mbps’ which controls how fast packets are sent. If defaults to a pretty low value so if you have a Gbps connection it will need to be specified to go fast.
> Do you have benchmarks for the total round trip time as well, rom sending to full reconstruction?
One use case I can talk a bit about involved transforming a large (100s of millions of records) hash database to individual S3 objects. Each was around 2kb. It took eons, even with Lambda executions sharing the work at scale.
The full end-to-end time can be observed using ‘—confirm FILE’ which will make raptor wait until the multipart upload is completed (and confirmation notification received). The baseline number is around .5s but it varies. Larger blocks with repair packets in use take longest.
> People can tar + ztsd infinitely many files and upload that archive normally and avoid the small-file problem altogether.
Sure, is that works for the use case. But sometimes small files i S3 are the desired outcome. For example, to serve as web pages or a static JSON API.
> Your pipeline involves quite a lot of moving parts.
It seems simple to me. It’s just a couple queues and a couple Lambda functions (one very simple). The FIFO queue is important and does double duty - ensuring only one completion event gets through, and keeping them in order. I didn’t add much error recovery. Resending files is often the fastest path. There are some ways to get there though. If you use the ‘—no-prefix’ flag only incomplete blocks will be processed on repeated uploads — not a solution, but something that could be built on.
> Pardon my dumb question. The final multi-part upload in S3 still uses the same number of HTTPs connections as if the file were sent from clients, right?
Yes, you have this right. Since we don’t control the S3 API all we can do is show how much better it could be if S3 had an API like this. It could save an insane amount of electricity.