In HN style, I'm going to diverge from the content and rant about the company:
Nanit needs this storage because they run cloud based baby cameras. Every Nanit user is uploading video and audio of their home/baby
live to Nanit without any E2EE. It's a hot mic sending anything you say near it to the cloud.
Their hardware essentially requires a subscription to use, even though it costs $200/camera. You must spend an additional $200 on a Nanit floor stand if you want sleep tracking. This is purely a software limitation since there's plenty of other ways to get an overhead camera mount. (I'm curious how they even detect if you're using the stand since it's just a USB-C cable. Maybe etags?)
Of course Nanit is a popular and successful product that many parents swear by. It just pains me to see cloud based in-home audio/video storage being so normalized. Self-hosted video isn't that hard but no one makes a baby-monitor centric solution. I'm sure the cloud based video storage model will continue to be popular because it's easy, but also because it helps justifies a recurring subscription.
edit: just noticed an irony in my comment. I'm ranting about Nanit locking users into their 3rd party cloud video storage, and the article is about Nanit's engineering team moving off a 3rd party (S3) and self-hosting their own storage. Props to them for getting off S3.
As a happy customer, I picked nanit because it actually worked. We didn’t even use the “smart” features, but “you can turn on the app from anywhere you happen to be and expect the video feed to work” is unfortunately a bar that no competitor I tried could meet. The others were mostly made by non-software companies with outsourced apps that worked maybe 50% of the time.
I wish we could have local-first and e2ee consumer software for this sort of thing, but given the choice of that or actually usable software, I am going to pick the latter.
I self host my "baby monitor" with UniFi Protect on UCG-Max and a G6 Instant wireless camera. It's more work to setup, but pretty easy for a techie. It has the "turn on the app anywhere and it works" feature, and with a 2TB SSD I get a month+ of video storage. Because storage is local, it doesn't need to compress the video and I get a super clear 4K image. And I use Homebridge to expose the camera over Apple HomeKit which is a convenient and a more user friendly way to access it. And HomeKit also gives you out-of-home access with a hub. I love my setup, but I couldn't in good conscience recommend it to a non-techie friend, especially if they're sleep deprived from their infant.
But I do miss the lack of any baby-specific features like sleep tracking. It has support for crying detection, but that's it.
I have a smarthome setup I built myself using Lua and a Raspberry Pi. Anything it can do locally can be securely exposed on the internet via a service like netbird , which I use for free and is literally a command to get running, or tail-scale which I believe is harder to use. I don’t have video but I think that would work in that scenario as well.
It's pretty cool! But homebridge is another service to run in a Docker container.. so even less user friendly. But it's definitely the primary way everyone that's not me accesses the baby camera. The out-of-home access requires a "HomeKit Hub" which can just be an Apple TV that's always plugged in. And HomeKit also has "HomeKit Secure Video" feature which is cloud based video storage, but with E2EE. But don't recommend their video storage really.
I have a bunch of cameras from various vendors, some with open FW, some with their original FW, all cut off from the internet. They used to be connected to Frigate but due to performance issues I offloaded the work to Scrypted on a RPi and an AppleTV and the setup works great. It was easy to set up and it's easier to use than any other app, assuming you are into the Apple Home ecosystem.
It's not really self hosted since it relies on Apple but it's the least evil at this point. Giving unencrypted video and audio to some company (if what OP says is right) would be way beyond my risk tolerance point.
Alternatively you can setup a vpn with rules that automatically enable vpn when you try to connect to specific addresses. Works with Tailscale and on-demand VPN for me. This will work with any IP webcam.
What competitor have you actually tried? My girlfriend’s parents have a few cheap TPlink solar powered CCTV and they work flawlessly since setup. I used to jerryrig an Android phone for Alfred and that too worked well.
I tried a high end Philips one and a Nest camera. Both were way less reliable than the Nanit. Possibly because they didn’t play nicely with my mesh WiFi at home. But regardless I just wanted to vouch for Nanit’s software, whatever they are doing with their networking and UX is really good.
They've mostly sold off bits of themselves, and/or licensed their name to other producers. It's highly unlikely that Philips actually made that camera.
Their networking is awful in my experience. The WiFi chip is cheap crap, extremely sensitive, cuts out a lot, and doesn’t support WPA3.
I had to set up a dedicated Nanit-only AP in my house in order to stabilize the connection. It would not work any other way, tried many different configurations, even other APs.
I have 2 free-roaming rabbits in one room of the house, we've been using Eufy camera to access live feed and found no issues with it, definitely would buy again. And the SD card recording allows us to seek a couple days into the past - it is pretty fun to watch the rabbits scramble to the automatic feeder at the set time.
My £15 TP-Link camera that we use as a baby monitor works 100% of the time. I can use it completely locally too with nothing sent to their servers at all, or use it through the internet if I want to. Got 4+ years of continuous use and counting, with zero issues.
The vtech camera is working well enough for me for what it’s worth. But any such app solution generally implies transfer through the company’s servers.
Yeah that’s fair, we had one of those too which absolutely did everything it advertised. The nanit is a different product that doubles as a home camera that lets you monitor your home while you’re away. Its software/networking is impressively reliable.
> Every Nanit user is uploading video and audio of their home/baby live to Nanit without any E2EE. It's a hot mic sending anything you say near it to the cloud.
Your way of phrasing it makes it sound like it would be fine to upload the video if it were end-to-end-encrypted. I think this is worth clarifying (since many don’t really understand the E2EE trade-off): E2EE is for smart clients that do all the processing, plus dumb servers that are only used for blind routing and storage. In this instance, it sounds like Nanit aren’t doing any routing or (persistent) storage: the sole purpose of the upload is offloading processing to the cloud. Given that, you can have transport encryption (typically TLS), but end-to-end encryption is not possible.
If you wanted the same functionality with end-to-end encryption, you’d need to do the video analysis locally, and upload the results, instead of uploading the entire video. This would presumably require more powerful hardware, or some way of offloading that to a nominated computer or phone.
Technically there are two clients: The camera and whatever device is used to access the feed.
I can absolutely imagine an architecture where video can be streamed in an encrypted manner, or stored in encrypted time-stamped blobs, allowing the server to provide rough searching, and then the client can perform fine-grained scanning.
This obviously doesn't enable any kind of processing of the video data on the server side, and doing it on the receiving client would require the feed to be active This means that any kind of processing would almost necessarily have to happen on the sending device, which would probably increase the power and compute requirements by a lot.
No, this doesn't get at the point of end-to-end encryption. Better to look at it in terms of the parties involved -- E2EE implies that there are two or more parties, and that only some of those parties should have unencrypted access.
In the case in point, the parent (camera owner) is one party and Nanit is another party. (Prior to the work in the linked post, AWS S3 was another party). The goal of E2EE is to deny plaintext access to some of these parties. So, in an E2EE deployment, Nanit (and AWS) would not have unencrypted access to the video content, even though they're storing it.
As chrismorgan pointed out, if Nanit did not have access to the unencrypted data, they could not do server-side video processing.
(Also, FWIW, there are multiple clients in this scenario -- the parents' phones are clients, and need unencrypted access to the video stream.)
(As an aside, where I used to work, we did some cool stuff with granting conditional access to certain server-side subsystems, so that the general data flow was all end-to-end encrypted, but customers could allow certain of our processes to be "ends" and have key access. This was really elegant; customers could dial in the level of server-side access that we had, and could see via the key authorization metadata which services had that access.)
I actually don’t really get the point of a cloud service for this. Aren’t babies usually left in situations where there’s at least one trusted adult locally available?
The "point" of the cloud service is that it's sadly usually the easiest way to create a [on-premise-device]<->[user's smartphone/laptop] for B2C/residential deployments of appliances (like the baby monitor in this case).
It's much easier to create a device<->internet connection + a smartphone<->internet connection that it is to deal with the myriad of issues that occur if you try to do local device<->smartphone connections in networks with unknown topology and quirks (e.g. ISP provider being overly conservative in their firewall presets). If that in general would be a more trivial issue you would see less cloud services.
(You would probably still a similar amount of cloud services due the increased monetization options, but this would level the playing field for local-only options.)
Hopefully as IPv6 gets more adoption we'll be able to open up some of these services, although IoT devices never get firmware updates so they'll have to sit behind Wireguard so maybe not.
Leading cause of death under one year is sudden infant death syndrome which happens mostly at nap time, situations where the adult may need rest, self care or housekeeping. You cannot fathomly watch an infant 24/7 especially if one parent is working and there's minimal support sistem (living far from relative, working grandparents etc)
So, 24/7 kinda, yeah... Realistically, the risk is relatively low I'd say, so to still stay a functioning parent with other duties (for baby or otherwise), you don't look 24/7
Yeah but the reality of the microSD card is weird. E.g. Eufy puts the video on the card but encrypts it so you have to pull it through the camera through the app to your phone.
It's hilariously crazy but we were given the cams as a gift so we stuck with them.
That's always annoyed me about Eufy, but it hasn't been a practical problem given they're mounted in hard-to-reach areas. I think the feature is to avoid a thief being able to view the footage. I like that they support RTSP access for a NAS/live viewing without their bloated app.
My parents bought a camcorder in 1995 and "self-hosted" the video just fine. But you're right it shouldn't even be something consumers should consider, because it should be the default and should be easy. You can get low power SSD-powered NAS devices now so hopefully this will change soon.
The baby monitor could have its own SD card and webserver and then you provide a smartphone app which uses local network discovery to find the server and talk to it.
In that case no parent needs to know about Synology or even IP addresses.
There is no technical requirement for an easy-to-use baby monitor to be cloud-connected. If there is no easy-to-use baby monitor which is not cloud-connected, that is a market problem, not a technical problem.
It's more that a typical parent has not thought of the need to have a baby monitor, until they have a baby (in which case, they're too busy to build out their own baby monitor stack).
Pay money to solve a problem and time-save as a parent is a valid business idea/strategy. The externalities that the parents might suffer if these businesses do not completely adhere to good security practices don't seem to come back to bite them (and most parents get lucky and not have any bad consequences - yet).
Maybe you want it to be easy to grant a babysitter access to the cameras temporarily and not bother getting them VPN'en into your CCTV network.
Maybe you want to check up on the babysitter (as creepy as that sounds, there might be good reasons). Or you're traveling but your partner is home, and you want to be able to see your sleeping child from half a world away.
I do think we've gone to far in the direction of cloud-only, but I don't think it's a bad option of have. The problem I have is that many of the companies running these services have really terrible security. So for S3 for a nanny cam, I'd assume that each customer have their own bucket, with their own credentials, but I doubt that's the case.
We've used an offline Infant Optics baby camera for three kids and have never wished for any of the smart features that online cameras offer. You really just want to know whether they are asleep and when they are crying. I just don't see a good use case for recording all that video for most kids. (I'm sure there are special needs situations where it is helpful)
> Self-hosted video isn't that hard but no one makes a baby-monitor centric solution
It sounds like they're not hosting it though. They are processing it, and storing it temporarily while it's queued.
A fully self hosted AI powered baby monitor that accurately detects sleep states and danger situations would be incredibly expensive today. Maybe not in a few years though.
We just used ipcams with our kids. Now with ubiquity it is dead simple to setup also storage for it. I think synology supports anything that emits rtsp.
Baby monitors around here -Alecto is a popular brand - cost twice as much and have only half the capabilities.
> Self-hosted video isn't that hard but no one makes a baby-monitor centric solution.
It's not that easy. The only usecase that is actually really fucking easy is when both the camera and the device trying to access it is in the same network - broadcasts for discovery, that's it. Although I've seen people turn on "client isolation" in their wifi back when I did computer repairs, so it's not a given that this works!
But as soon as that assumption goes out the window - and if it's just you going into the garden to check on some weeds where the wifi doesn't reach - the task suddenly becomes so, so much harder:
- the "easiest" case is an ISP that hands your wifi router a globally routed IPv4 address, allows UPnP to be configured, and the user has UPnP configured. All that the camera has to do here is to request a port opening and that's it. Still, you as manufacturer need a server to store a mapping between user, IP address and port. (And you need to hope that the user's mobile device or their ISP doesn't have a nasty firewall blocking non-standard ports)
- No UPnP? Now you as manufacturer either need some STUN/TURN server or explain to the user how to manually enable port forwarding.
- Worst case: the user's ISP either has IPv6 only, CGNAT, double/triple/... NAT or similar shit in play because they don't have enough IP addresses to supply to their customer base. That's pretty much impossible even with STUN/TURN, sooo many ways for things to go wrong along the path.
- even a theoretical fully IPv6 world where everyone has globally routed IPv6 addresses everywhere and all ISPs have their routing working still wouldn't solve the issue... because consumer ISP routers enable a firewall on IPv6 to avoid stuff like "online game cheaters 0wning their opponents running an outdated version of their game".
The sad reality is, running a cloud service is the only actually pain-free way for any given smart Thing to work as the customer expects it.
And on top of that, a NAS capable of storing video costs about 300-ish bucks with a HDD capable of running 24/7 and eats about 10-ish watts of electricity, which is quite the cost factor on its own.
Sure, the "nerd population" here on HN can rig something up that works in a matter of a few days, including some rudimentary AI to spot if the baby managed to escape the crib. But the 99% of people out there will crash at the "please open your router's config page to allow UDP port 65535 passthrough" step, if only because they forgot the password that they set five years ago.
Of course you don't _need_ it, but it's a useful convenience. Due to the layout of our house it was quite hard to hear my toddler if he was crying in the middle of the night - we often wouldn't wake up to it. So the monitor was very helpful.
Why on earth do you need an app and a camera? The same basic VTech audio monitors that are basically the same for many decades now work great, don't cost a fortune and there's no question of "where is this data going?" It's all just a big cash grab for people who need chincy tech toys for a non-problem that's better solved with way more simple kit.
I used to work with my laptop, sitting near my baby. Also, I used a timer to follow 45m sleep patterns, so technically there’s no need to react to anything within first 45m, but most times first 1h30m (45+45m).
Yeah, so now you're basically running a heavy instance in order to get the network throughput and the RAM, but not really using that much CPU when you could probably handle the encode with the available headroom. Although the article lists TLS handshakes as being a significant source of CPU usage, I must be missing something because I don't see how that is anywhere near the top of the constraints of a system like this.
Regardless, I enjoyed the article and I appreciate that people are still finding ways to build systems tailored to their workflows.
They didn’t actually do what the headline claims. They made a memory cache which sits in front of S3 for the happy path. Cool but not nearly rolling your own S3
My first thought is, why bother with local storage if your turnaround on video chunks is 2 seconds? What's disk going to add besides a little bit more resiliency in that 2 second time frame? This at the cost of having slower pod startups given you have to mount the PVC, and a small performance hit of writing to a filesystem instead of memory.
All moot anyway given that the cameras/proxy allegedly has retries built-in, but interested to hear your thoughts.
This may be an obvious point, but I didn't see it mentioned in the (otherwise excellent) article: I would have been interested in the cost saving in just implementing the 'delete on read' with S3 that they ended up using with the home-made in-memory cache solution. I can't see this on the S3 billing page, but if the usage is billed per-second, as with some other AWS services, then the savings may be significant.
The solution they document also matches the S3 'reduced redundancy' storage option, so I hope they had this enabled from day one.
I suspect it's a massive amount, as S3 is one of the cheaper services. As we evaluate moving all of our compute off of AWS, S3 (and SQS) are probably services we'll retain because they are still amazing values.
Exactly, my first thought was "Why in earth would anyone think that S3 was the right service to store millions of tiny ephemeral files?" and now it seems they have invented their own in-memory store instead of just using something like Redis. I also wonder what happens if their DIY thingy crashes, are the videos lost? Why not send to Kinesis or SQS in the first place?
I made my own S3 as well. I used two S3-compatible services before but there was always some issue(first one failed to upload certain file, no matter what and support was unhelpful; second one did not migrate with file metadata properly so i knew this would be ongoing problem). In the end, it is just a dumb file store, nothing else. All you need to do is to write a basic HTTPS API layer and some logic to handle database for the file metadata and possibly location. That is about it. Takes a few days with testing.
But then you also have to think about file uploads and file downloads. You cannot have a single server fulfilling all the roles, otherwise you have a bottleneck.
So this file storage became a private backend service that end-users never access directly. I have added upload services, whose sole purpose is to allow users to upload files and only then upload them to this central file store, essentially creating a distributed file upload queue(there is also a bit more logic regarding file id creation and validation).
Secondly, own CDN was needed for downloads. But only because I use custom access handling and could not have used any of the commercial services(though they do support access via tokens, it just was not working for me). This was tricky because I wanted for the nodes to distribute files between themselves and not always fetch them from the origin to avoid network costs on the origin server. So they had to find each other, talk to each other and know who has which file.
In short, rolling your own is not as hard as it might seem and should be preferable. Maybe to save time, use cloud at the beginning, but once you are up and running and your business idea is validated by having customer, immediately move to your own infra in order to avoid astronomical costs of cloud services.
btw, i also do video processing like mentioned in the blog post :)
I’m sufficiently old / sensible (you decide) to think that uploading video of your baby (to anywhere) is fucking weird and fucking spooky and not needed anyway. This is a solution that doesn’t have a problem. Worse: it prays on parental / young parental fears. There’s nothing here - this is not a product that’s needed. You don’t need to “track” your baby, ffs. You don’t need to watch it while it sleeps. You don’t need “every breath, seen”. People have been having babies for fucking centuries without entering them into this hyper weird surveillance state at birth.
What an appalling screwed up world we seem to have manufactured for ourselves.
Of all the hills to die on w.r.t. how the world is screwed up, this seems like the silliest.
Different folks parent differently, culture evolves. You're free to have your "old school" thoughts as are people who use services like this.
Its not like they're publishing it in public. The service in discussion especially just stores it in the server only temporarily to use ML to detect things such as sleeping or crying. Sounds innocuous to me.
Many of us can do the math ourselves and choose to make choices based on our own beliefs. That's true freedom.
Oh, I’ve got many, many hills to die on. But at some level this is to me symptomatic of two broad things - maybe these are bigger hills:
1) a business making (seemingly) huge profits from the fears of others; more specifically from the fears of young parents who are quite often vulnerable. And, I’d suggest, offering a “solution” with no problem attached to it apart from that fear. Not to mention the issues around privacy, the fact that however many “it’s safe and encrypted” services get hacked, or sold on when the IPO comes around.
2) As another commenter says below - this is symptomatic of a type of parenting which ultimately creates fearful, anxious, badly adjusted children. Study after study shows that “free roam” children end up as better balanced humans. I’m not saying that watching and monitoring and stat-ifying your newborn is immediately going to make them anxious, but it says something about parenting which to me is unhealthy, obsessive, and ultimately not about freedom for the most important beings here: the kids.
I’m extrapolating of course, but hopefully my drift is somewhat clear..
> Different folks parent differently, culture evolves.
You are framing it as if every change is in a positive direction, which it clearly isn't. Risking at sounding like an old man yelling at clouds, look at the kids these days. They are so dependent, and so sensitive to negative stimuli or emotions.
Parents need to realise that they need to grow adults, not perpetual kids.
> look at the kids these days. They are so dependent, and so sensitive to negative stimuli or emotions.
I think many of us have found people writing comments like this are not interacting with children very much. More just reading the takes of others who also don't interact with children.
And if this was a legitimate problem to address, you would not address it by taking away baby monitors.
The topic of the conversation widened a bit from baby monitors. I of course do not advocate that taking away baby monitors will fix our dilapidated societies.
And while it is certainly true that I don't interact with young children a lot, the case is different for older ones or "young adults".
This feels like they were using the wrong architecture from the start, and are now papering over that problem with additional layers of cache.
The only practical reason to put a video in S3 for an average of 2 seconds is to provide additional redundancy, and replacing that with a cache removes most of the redundancy.
Feels like if you uploaded these to an actual server, the server could process them on upload, and you could eliminate S3, the queue in SQS, and the lambdas all in one fell swoop...
> I'm curious how many engineers per year this costs to maintain
The end of the article has this:
> Consider custom infrastructure when you have both: sufficient scale for meaningful cost savings, and specific constraints that enable a simple solution. The engineering effort to build and maintain your system must be less than the infrastructure costs it eliminates. In our case, specific requirements (ephemeral storage, loss tolerance, S3 fallback) let us build something simple enough that maintenance costs stay low. Without both factors, stick with managed services.
And I am curious how many engineer years it requires to port code to cloud services and deal with multiple issues you cannot even debug due to not having root privileges in the cloud.
Without cloud, saving a file is as simple as "with open(...) as f: f.write(data)" + adding a record to DB. And no weird network issues to debug.
> as simple as "with open(...) as f: f.write(data)"
Save where?
With what redundancy?
With what access policies?
With what backup strategy?
With what network topology?
With what storage equipment and file system and HVAC system and...
Without on-prem, saving a file is as simple as s3.put_object() !
>> Without cloud, saving a file is as simple as "with open(...) as f: f.write(data)" + adding a record to DB.
> Save where? With what redundancy? With what access policies? With what backup strategy? With what network topology? With what storage equipment and file system and HVAC system and...
Most of these concerns can be addressed with ZFS[0] provided by FreeBSD systems hosted in triple-A data centers.
> Save where? With what redundancy? With what access policies? With what backup strategy? With what network topology? With what storage equipment and file system and HVAC system and...
Wow that's a lot to learn before using s3... I wonder how much it costs in salaries.
> With what network topology?
You don't need to care about this when using SSDs/HDDs.
> With what access policies?
Whichever is defined in your code, no restrictions unlike in S3. No need to study complicated AWS documentation and navigate through multiple consoles (this also costs you salaries by the way). No risk of leaking files due to misconfigured cloud services.
> With what backup strategy?
Automatically backed up with rest of your server data, no need to spend time on this.
>> No risk of leaking files due to misconfigured cloud services.
> One misconfigured .htaccess file for example, could result in leaking files.
I don't think you are making a compelling case here, since both scenarios result in an undesirable exposure. Unless your point is both cloud services and local file systems can be equally exploited?
It sounds like you’re not at the scale where cloud storage is obviously useful. By the time you definitely need S3/GCS you have problems making sure files are accessible everywhere. “Grep” is a ludicrous proposition against large blob stores
I inherited an S3 bucket where hundreds of thousands of files were written to the bucket root. Every filename was just a uuid. ls might work after waiting to page though to get every file. To grep you would need to download 5 TB.
It's probably going to be dog slow. I dealt with HDDs where just iterating through all files and directories takes hours, and network storage is going to be even slower at this scale.
You can't ever definitively answer most of those questions on someone else's cloud. You just take Amazons word for whatever number of nines they claim it has.
Bro were you off grid last week. Your questions equally apply to AWS, you just magically handwave away all those questions as if AWS/GCP/Azure outages aren’t a thing.
Ah that is where logging and traceability comes in! But not to worry, the cloud has excellent tools for that! The fact that logging and tracing will become half your cloud cost, oh well let's just sweep that under the rug.
> Without cloud, saving a file is as simple as "with open(...) as f: f.write(data)" + adding a record to DB. And no weird network issues to debug.
There may be some additional features that S3 has over a direct filesystem write to a SSD in your closet. The people paying for cloud spend are paying for those features.
What I notice, that large companies use their own private cloud and datacenters. At their scale, it is cheaper to have their own storage. As a side business, they also sell cloud services themselves. And small companies probably don't have that much data to justify paying for a cloud instead of buying several SSDs/HDDs or creating SMB share on their Windows server.
So, you want a place to store many files in a short period of time and when there's a new file, somebody must be notified?
Have you ever thought of using a postgresql db (also on aws) to store those files and use CDC to publish messages about those files to a kafka topic? In your original way, we need 3 aws services: s3, lambda and sqs. With this way, we need 2: postgresql and kafka. I'm not sure how well this method works though :-)
I’m mostly just impressed that some janky baby monitor has racked up server fees on this scale. Amazing example of absolutely horrible engineering.
Also, just take an old phone from your drawer full of old phones, slap some free camera app on it, zip tie a car phone mount to the crib, and boom you have a free baby monitor.
S3 certainly saves a lot of hassle, but in certain use cases, it really is prohibitively expensive.
Has anyone tried self-hosted alternatives like MinIO or SeaweedFS? Or taken even more radical approaches?
How do you balance between stability, maintenance overhead, and cost savings?
MinIO has moved away from having a free community fork, and I think it's base cost is close to $100k a year. I've been using Garage and been happy, but as a single dev and orders of magnitude smaller than the OP, so there are certainly edge cases I'm missing to compare the two.
I'm a fellow new Garage user. I have had a great time so far - but I also don't need much. My use case is to share data analysis results with a small team. I wanted something simple to manage that can provide an s3 like interface to work with off the shelf data analysis tools.
Video processing is one of those things that need caution when doing serverlessly. This solution makes sense, especially because S3s durability guarantees aren't needed.
Their architecture is internet bandwidth heavy and storage heavy; these are some of the most expensive things in AWS. You probably want to use a different provider for those things.
> It turns out that when AWS says an instance can do “Up to 12.5 Gbps”, that’s burstable networking backed by credits; when you’re below the baseline, you accrue credits and can burst for short periods.
Yes, AWS has a burst rating and a sustained/baseline rating for both EBS types as well as instance types. Use https://instances.vantage.sh/ (and make sure you choose specific columns) to compare specific criteria and then export as a CSV to find the lowest price that matches your performance/feature/platform criteria. Design to the baseline if you need guaranteed performance. Do sustained performance testing.
> When we Terminated connections idle >10 minutes, memory dropped by ~1 GB immediately; confirming the leak was from dangling sockets. Fix: make sockets short-lived and enforce time limits.
We used to do that with Apache 20 years ago. Config option forces a forked subchild to exit after N requests to avoid the inevitable memory leaks. AKA the Windows 95 garbage collector (a reboot a day keeps the slowness at bay).
FWIW, if the business feasibility of your architecture depends on custom stuff, performance enhancements, etc, you will find that you eventually have harder and harder problems to solve to keep your business functioning. It's more reliable to waste money on a solution that is brainless, than invest human ingenuity/technical mastery in a solution that is frugal.
They're very ingest heavy compared to how much of it is actually streamed out and to a very small/local audience so probably don't even need a cdn. And ingest on aws is free.
On the other hand, S3 is kind of ridiculously expensive compared to even more expensive on-prem options like a PureStorage SSDs array. With spindles on Ceph you can probably get quite a bit lower than AWS's 2c/Gig/mo. Or you can just use R2 with colocated servers for ingest.
Storing the data in a foreign cloud would allow foreign nation to play funny tricks on the country. What they need is not the cloud but sane backup system.
I mean the "S3" could be replaced with object storage. I guess thats the technical term anyway. Having said that just goes to show how cheap S3 is, if after all of this, the savings are just $500k. Definitely money saved but not a lot.
In HN style, I'm going to diverge from the content and rant about the company:
Nanit needs this storage because they run cloud based baby cameras. Every Nanit user is uploading video and audio of their home/baby live to Nanit without any E2EE. It's a hot mic sending anything you say near it to the cloud.
Their hardware essentially requires a subscription to use, even though it costs $200/camera. You must spend an additional $200 on a Nanit floor stand if you want sleep tracking. This is purely a software limitation since there's plenty of other ways to get an overhead camera mount. (I'm curious how they even detect if you're using the stand since it's just a USB-C cable. Maybe etags?)
Of course Nanit is a popular and successful product that many parents swear by. It just pains me to see cloud based in-home audio/video storage being so normalized. Self-hosted video isn't that hard but no one makes a baby-monitor centric solution. I'm sure the cloud based video storage model will continue to be popular because it's easy, but also because it helps justifies a recurring subscription.
edit: just noticed an irony in my comment. I'm ranting about Nanit locking users into their 3rd party cloud video storage, and the article is about Nanit's engineering team moving off a 3rd party (S3) and self-hosting their own storage. Props to them for getting off S3.
As a happy customer, I picked nanit because it actually worked. We didn’t even use the “smart” features, but “you can turn on the app from anywhere you happen to be and expect the video feed to work” is unfortunately a bar that no competitor I tried could meet. The others were mostly made by non-software companies with outsourced apps that worked maybe 50% of the time.
I wish we could have local-first and e2ee consumer software for this sort of thing, but given the choice of that or actually usable software, I am going to pick the latter.
I self host my "baby monitor" with UniFi Protect on UCG-Max and a G6 Instant wireless camera. It's more work to setup, but pretty easy for a techie. It has the "turn on the app anywhere and it works" feature, and with a 2TB SSD I get a month+ of video storage. Because storage is local, it doesn't need to compress the video and I get a super clear 4K image. And I use Homebridge to expose the camera over Apple HomeKit which is a convenient and a more user friendly way to access it. And HomeKit also gives you out-of-home access with a hub. I love my setup, but I couldn't in good conscience recommend it to a non-techie friend, especially if they're sleep deprived from their infant.
But I do miss the lack of any baby-specific features like sleep tracking. It has support for crying detection, but that's it.
This is indeed far more of a "HN Style" comment.
Not enough “anyone can set it up trivially“.
You've still had to buy a proprietary system, it just happens to run locally? Not really much better is it.
Sure, not perfect, but quite a bit better. Getting from A to Z involves a few letters inbetween...
I came here to say, this is exactly what I do also.
Unifi accidentally made a fantastic baby monitor.
The recent APIs they’ve built makes me hopeful that I could run an AI model against the footage eventually and build those Ai features for myself.
Ok that’s really cool; I didn’t know you could set up Apple’s smart home thingy to forward a live feed to the cloud.
I have a smarthome setup I built myself using Lua and a Raspberry Pi. Anything it can do locally can be securely exposed on the internet via a service like netbird , which I use for free and is literally a command to get running, or tail-scale which I believe is harder to use. I don’t have video but I think that would work in that scenario as well.
It's pretty cool! But homebridge is another service to run in a Docker container.. so even less user friendly. But it's definitely the primary way everyone that's not me accesses the baby camera. The out-of-home access requires a "HomeKit Hub" which can just be an Apple TV that's always plugged in. And HomeKit also has "HomeKit Secure Video" feature which is cloud based video storage, but with E2EE. But don't recommend their video storage really.
I have a bunch of cameras from various vendors, some with open FW, some with their original FW, all cut off from the internet. They used to be connected to Frigate but due to performance issues I offloaded the work to Scrypted on a RPi and an AppleTV and the setup works great. It was easy to set up and it's easier to use than any other app, assuming you are into the Apple Home ecosystem.
It's not really self hosted since it relies on Apple but it's the least evil at this point. Giving unencrypted video and audio to some company (if what OP says is right) would be way beyond my risk tolerance point.
Alternatively you can setup a vpn with rules that automatically enable vpn when you try to connect to specific addresses. Works with Tailscale and on-demand VPN for me. This will work with any IP webcam.
What competitor have you actually tried? My girlfriend’s parents have a few cheap TPlink solar powered CCTV and they work flawlessly since setup. I used to jerryrig an Android phone for Alfred and that too worked well.
My impression is live feed is a solved problem.
I tried a high end Philips one and a Nest camera. Both were way less reliable than the Nanit. Possibly because they didn’t play nicely with my mesh WiFi at home. But regardless I just wanted to vouch for Nanit’s software, whatever they are doing with their networking and UX is really good.
Beware of Philips in general: https://www.youtube.com/watch?v=WE58YisgFeQ
They've mostly sold off bits of themselves, and/or licensed their name to other producers. It's highly unlikely that Philips actually made that camera.
Their networking is awful in my experience. The WiFi chip is cheap crap, extremely sensitive, cuts out a lot, and doesn’t support WPA3.
I had to set up a dedicated Nanit-only AP in my house in order to stabilize the connection. It would not work any other way, tried many different configurations, even other APs.
i have a tplink as well and can vouch for it. it has iphone and android apps and can show live feed. mine costed 30$ and free live video.
I have 2 free-roaming rabbits in one room of the house, we've been using Eufy camera to access live feed and found no issues with it, definitely would buy again. And the SD card recording allows us to seek a couple days into the past - it is pretty fun to watch the rabbits scramble to the automatic feeder at the set time.
My £15 TP-Link camera that we use as a baby monitor works 100% of the time. I can use it completely locally too with nothing sent to their servers at all, or use it through the internet if I want to. Got 4+ years of continuous use and counting, with zero issues.
The vtech camera is working well enough for me for what it’s worth. But any such app solution generally implies transfer through the company’s servers.
It seems possible to establish a p2p connection with the camera where the company servers act as a broker.
Yeah that’s fair, we had one of those too which absolutely did everything it advertised. The nanit is a different product that doubles as a home camera that lets you monitor your home while you’re away. Its software/networking is impressively reliable.
> Every Nanit user is uploading video and audio of their home/baby live to Nanit without any E2EE. It's a hot mic sending anything you say near it to the cloud.
Your way of phrasing it makes it sound like it would be fine to upload the video if it were end-to-end-encrypted. I think this is worth clarifying (since many don’t really understand the E2EE trade-off): E2EE is for smart clients that do all the processing, plus dumb servers that are only used for blind routing and storage. In this instance, it sounds like Nanit aren’t doing any routing or (persistent) storage: the sole purpose of the upload is offloading processing to the cloud. Given that, you can have transport encryption (typically TLS), but end-to-end encryption is not possible.
If you wanted the same functionality with end-to-end encryption, you’d need to do the video analysis locally, and upload the results, instead of uploading the entire video. This would presumably require more powerful hardware, or some way of offloading that to a nominated computer or phone.
In other words, E2EE requires two or more clients, and only on these clients the information is in clear.
In the case of this product, there is only one client (and a server).
E2EE bills then down to having the traffic encrypted like you have with a https website.
Technically there are two clients: The camera and whatever device is used to access the feed.
I can absolutely imagine an architecture where video can be streamed in an encrypted manner, or stored in encrypted time-stamped blobs, allowing the server to provide rough searching, and then the client can perform fine-grained scanning.
This obviously doesn't enable any kind of processing of the video data on the server side, and doing it on the receiving client would require the feed to be active This means that any kind of processing would almost necessarily have to happen on the sending device, which would probably increase the power and compute requirements by a lot.
No, this doesn't get at the point of end-to-end encryption. Better to look at it in terms of the parties involved -- E2EE implies that there are two or more parties, and that only some of those parties should have unencrypted access.
In the case in point, the parent (camera owner) is one party and Nanit is another party. (Prior to the work in the linked post, AWS S3 was another party). The goal of E2EE is to deny plaintext access to some of these parties. So, in an E2EE deployment, Nanit (and AWS) would not have unencrypted access to the video content, even though they're storing it.
As chrismorgan pointed out, if Nanit did not have access to the unencrypted data, they could not do server-side video processing.
(Also, FWIW, there are multiple clients in this scenario -- the parents' phones are clients, and need unencrypted access to the video stream.)
(As an aside, where I used to work, we did some cool stuff with granting conditional access to certain server-side subsystems, so that the general data flow was all end-to-end encrypted, but customers could allow certain of our processes to be "ends" and have key access. This was really elegant; customers could dial in the level of server-side access that we had, and could see via the key authorization metadata which services had that access.)
I think the point is that effectively this is E2EE due to TLS, because the server is expected to be able to decrypt the data (and so is one “end”).
That’s not what most people expect though.
I actually don’t really get the point of a cloud service for this. Aren’t babies usually left in situations where there’s at least one trusted adult locally available?
The "point" of the cloud service is that it's sadly usually the easiest way to create a [on-premise-device]<->[user's smartphone/laptop] for B2C/residential deployments of appliances (like the baby monitor in this case).
It's much easier to create a device<->internet connection + a smartphone<->internet connection that it is to deal with the myriad of issues that occur if you try to do local device<->smartphone connections in networks with unknown topology and quirks (e.g. ISP provider being overly conservative in their firewall presets). If that in general would be a more trivial issue you would see less cloud services.
(You would probably still a similar amount of cloud services due the increased monetization options, but this would level the playing field for local-only options.)
Hopefully as IPv6 gets more adoption we'll be able to open up some of these services, although IoT devices never get firmware updates so they'll have to sit behind Wireguard so maybe not.
Leading cause of death under one year is sudden infant death syndrome which happens mostly at nap time, situations where the adult may need rest, self care or housekeeping. You cannot fathomly watch an infant 24/7 especially if one parent is working and there's minimal support sistem (living far from relative, working grandparents etc)
"Measures not shown to be useful include positioning devices and baby monitors."
https://en.wikipedia.org/wiki/SIDS
Yes, the normal solution to wondering how a baby is doing is to look over at it.
24/7?
> There is usually no noise or evidence of struggle
from [0]
https://en.wikipedia.org/wiki/SIDS
So, 24/7 kinda, yeah... Realistically, the risk is relatively low I'd say, so to still stay a functioning parent with other duties (for baby or otherwise), you don't look 24/7
> Self-hosted video isn't that hard
Self-hosting video is not something the typical user of a baby monitor would ever even consider.
A microSD card in the camera, like most others use?
From the product description though it sounds like sleep analysis is what you're paying for, which they do on servers analyzing the video.
Yeah but the reality of the microSD card is weird. E.g. Eufy puts the video on the card but encrypts it so you have to pull it through the camera through the app to your phone.
It's hilariously crazy but we were given the cams as a gift so we stuck with them.
That's always annoyed me about Eufy, but it hasn't been a practical problem given they're mounted in hard-to-reach areas. I think the feature is to avoid a thief being able to view the footage. I like that they support RTSP access for a NAS/live viewing without their bloated app.
My parents bought a camcorder in 1995 and "self-hosted" the video just fine. But you're right it shouldn't even be something consumers should consider, because it should be the default and should be easy. You can get low power SSD-powered NAS devices now so hopefully this will change soon.
Extraordinary claims require extraordinary evidence.
I'm not leaving a baby at home while I go on vacation. I would never be on another network, even. Why need the cloud?
Because it’s easy and convenient for new parents.
The typical parent has never heard of Synology or Ubiquiti, doesn’t have a NAS, and gets whatever tech their ISP gave/rents them.
The baby monitor could have its own SD card and webserver and then you provide a smartphone app which uses local network discovery to find the server and talk to it.
In that case no parent needs to know about Synology or even IP addresses.
There is no technical requirement for an easy-to-use baby monitor to be cloud-connected. If there is no easy-to-use baby monitor which is not cloud-connected, that is a market problem, not a technical problem.
It's more that a typical parent has not thought of the need to have a baby monitor, until they have a baby (in which case, they're too busy to build out their own baby monitor stack).
Pay money to solve a problem and time-save as a parent is a valid business idea/strategy. The externalities that the parents might suffer if these businesses do not completely adhere to good security practices don't seem to come back to bite them (and most parents get lucky and not have any bad consequences - yet).
Maybe you want it to be easy to grant a babysitter access to the cameras temporarily and not bother getting them VPN'en into your CCTV network.
Maybe you want to check up on the babysitter (as creepy as that sounds, there might be good reasons). Or you're traveling but your partner is home, and you want to be able to see your sleeping child from half a world away.
I do think we've gone to far in the direction of cloud-only, but I don't think it's a bad option of have. The problem I have is that many of the companies running these services have really terrible security. So for S3 for a nanny cam, I'd assume that each customer have their own bucket, with their own credentials, but I doubt that's the case.
We've used an offline Infant Optics baby camera for three kids and have never wished for any of the smart features that online cameras offer. You really just want to know whether they are asleep and when they are crying. I just don't see a good use case for recording all that video for most kids. (I'm sure there are special needs situations where it is helpful)
Now imagine all that video/audio footage being used/sold for AI training data.
> Self-hosted video isn't that hard but no one makes a baby-monitor centric solution
It sounds like they're not hosting it though. They are processing it, and storing it temporarily while it's queued.
A fully self hosted AI powered baby monitor that accurately detects sleep states and danger situations would be incredibly expensive today. Maybe not in a few years though.
We just used ipcams with our kids. Now with ubiquity it is dead simple to setup also storage for it. I think synology supports anything that emits rtsp.
Baby monitors around here -Alecto is a popular brand - cost twice as much and have only half the capabilities.
> Self-hosted video isn't that hard but no one makes a baby-monitor centric solution.
It's not that easy. The only usecase that is actually really fucking easy is when both the camera and the device trying to access it is in the same network - broadcasts for discovery, that's it. Although I've seen people turn on "client isolation" in their wifi back when I did computer repairs, so it's not a given that this works!
But as soon as that assumption goes out the window - and if it's just you going into the garden to check on some weeds where the wifi doesn't reach - the task suddenly becomes so, so much harder:
- the "easiest" case is an ISP that hands your wifi router a globally routed IPv4 address, allows UPnP to be configured, and the user has UPnP configured. All that the camera has to do here is to request a port opening and that's it. Still, you as manufacturer need a server to store a mapping between user, IP address and port. (And you need to hope that the user's mobile device or their ISP doesn't have a nasty firewall blocking non-standard ports)
- No UPnP? Now you as manufacturer either need some STUN/TURN server or explain to the user how to manually enable port forwarding.
- Worst case: the user's ISP either has IPv6 only, CGNAT, double/triple/... NAT or similar shit in play because they don't have enough IP addresses to supply to their customer base. That's pretty much impossible even with STUN/TURN, sooo many ways for things to go wrong along the path.
- even a theoretical fully IPv6 world where everyone has globally routed IPv6 addresses everywhere and all ISPs have their routing working still wouldn't solve the issue... because consumer ISP routers enable a firewall on IPv6 to avoid stuff like "online game cheaters 0wning their opponents running an outdated version of their game".
The sad reality is, running a cloud service is the only actually pain-free way for any given smart Thing to work as the customer expects it.
And on top of that, a NAS capable of storing video costs about 300-ish bucks with a HDD capable of running 24/7 and eats about 10-ish watts of electricity, which is quite the cost factor on its own.
Sure, the "nerd population" here on HN can rig something up that works in a matter of a few days, including some rudimentary AI to spot if the baby managed to escape the crib. But the 99% of people out there will crash at the "please open your router's config page to allow UDP port 65535 passthrough" step, if only because they forgot the password that they set five years ago.
This is the reason I refused to buy Nanit cameras, instead opting for unconnected models. E2E encryption is table stakes.
By the way you dont need a video (or hell even audio) baby monitor. Source: 2 kids.
Of course you don't _need_ it, but it's a useful convenience. Due to the layout of our house it was quite hard to hear my toddler if he was crying in the middle of the night - we often wouldn't wake up to it. So the monitor was very helpful.
Why on earth do you need an app and a camera? The same basic VTech audio monitors that are basically the same for many decades now work great, don't cost a fortune and there's no question of "where is this data going?" It's all just a big cash grab for people who need chincy tech toys for a non-problem that's better solved with way more simple kit.
You misunderstand; we're on the same wavelength. I'm not talking about an app, I'm talking about a basic audio baby monitor.
Same here. I wonder if the market is for first-time parents and people who work 8+ hour days.
I used to work with my laptop, sitting near my baby. Also, I used a timer to follow 45m sleep patterns, so technically there’s no need to react to anything within first 45m, but most times first 1h30m (45+45m).
"Self-hosted video isn't that hard but no one makes a baby-monitor centric solution"
I don't understand this attitude, sure its easy for some people but MOST people want an easy out of the box solution
its nothing wrong with that
Tbh I feel this in one of those that would be significantly cleaner without serverless in first place.
Sticking something with 2 second lifespan on disk to shoehorn it into aws serverless paradigm created problems and cost out of thin air here
Good solution moving at least partially to a in memory solution though
Yeah, so now you're basically running a heavy instance in order to get the network throughput and the RAM, but not really using that much CPU when you could probably handle the encode with the available headroom. Although the article lists TLS handshakes as being a significant source of CPU usage, I must be missing something because I don't see how that is anywhere near the top of the constraints of a system like this.
Regardless, I enjoyed the article and I appreciate that people are still finding ways to build systems tailored to their workflows.
Maybe they’re not using keepalives in their clients causing thousands of handshakes per second?
Yes, they mention this as a 'fix' for connection-related memory usage:
> Disable keep-alive: close the connection immediately after each upload completes.
Very odd idea.
They didn’t actually do what the headline claims. They made a memory cache which sits in front of S3 for the happy path. Cool but not nearly rolling your own S3
I also didnt get why the cache had to be memory and not local name storage..
Out of curiosity, why?
My first thought is, why bother with local storage if your turnaround on video chunks is 2 seconds? What's disk going to add besides a little bit more resiliency in that 2 second time frame? This at the cost of having slower pod startups given you have to mount the PVC, and a small performance hit of writing to a filesystem instead of memory.
All moot anyway given that the cameras/proxy allegedly has retries built-in, but interested to hear your thoughts.
This may be an obvious point, but I didn't see it mentioned in the (otherwise excellent) article: I would have been interested in the cost saving in just implementing the 'delete on read' with S3 that they ended up using with the home-made in-memory cache solution. I can't see this on the S3 billing page, but if the usage is billed per-second, as with some other AWS services, then the savings may be significant.
The solution they document also matches the S3 'reduced redundancy' storage option, so I hope they had this enabled from day one.
They saved $500k on what total sum? $500'001 or 55'000'000? Without this info the post is moot.
I suspect it's a massive amount, as S3 is one of the cheaper services. As we evaluate moving all of our compute off of AWS, S3 (and SQS) are probably services we'll retain because they are still amazing values.
Sounds like the title should have been
> We used S3 even though it wasn’t the right service
Exactly, my first thought was "Why in earth would anyone think that S3 was the right service to store millions of tiny ephemeral files?" and now it seems they have invented their own in-memory store instead of just using something like Redis. I also wonder what happens if their DIY thingy crashes, are the videos lost? Why not send to Kinesis or SQS in the first place?
I made my own S3 as well. I used two S3-compatible services before but there was always some issue(first one failed to upload certain file, no matter what and support was unhelpful; second one did not migrate with file metadata properly so i knew this would be ongoing problem). In the end, it is just a dumb file store, nothing else. All you need to do is to write a basic HTTPS API layer and some logic to handle database for the file metadata and possibly location. That is about it. Takes a few days with testing.
But then you also have to think about file uploads and file downloads. You cannot have a single server fulfilling all the roles, otherwise you have a bottleneck.
So this file storage became a private backend service that end-users never access directly. I have added upload services, whose sole purpose is to allow users to upload files and only then upload them to this central file store, essentially creating a distributed file upload queue(there is also a bit more logic regarding file id creation and validation).
Secondly, own CDN was needed for downloads. But only because I use custom access handling and could not have used any of the commercial services(though they do support access via tokens, it just was not working for me). This was tricky because I wanted for the nodes to distribute files between themselves and not always fetch them from the origin to avoid network costs on the origin server. So they had to find each other, talk to each other and know who has which file.
In short, rolling your own is not as hard as it might seem and should be preferable. Maybe to save time, use cloud at the beginning, but once you are up and running and your business idea is validated by having customer, immediately move to your own infra in order to avoid astronomical costs of cloud services.
btw, i also do video processing like mentioned in the blog post :)
I’m sufficiently old / sensible (you decide) to think that uploading video of your baby (to anywhere) is fucking weird and fucking spooky and not needed anyway. This is a solution that doesn’t have a problem. Worse: it prays on parental / young parental fears. There’s nothing here - this is not a product that’s needed. You don’t need to “track” your baby, ffs. You don’t need to watch it while it sleeps. You don’t need “every breath, seen”. People have been having babies for fucking centuries without entering them into this hyper weird surveillance state at birth.
What an appalling screwed up world we seem to have manufactured for ourselves.
Of all the hills to die on w.r.t. how the world is screwed up, this seems like the silliest.
Different folks parent differently, culture evolves. You're free to have your "old school" thoughts as are people who use services like this.
Its not like they're publishing it in public. The service in discussion especially just stores it in the server only temporarily to use ML to detect things such as sleeping or crying. Sounds innocuous to me.
Many of us can do the math ourselves and choose to make choices based on our own beliefs. That's true freedom.
Oh, I’ve got many, many hills to die on. But at some level this is to me symptomatic of two broad things - maybe these are bigger hills:
1) a business making (seemingly) huge profits from the fears of others; more specifically from the fears of young parents who are quite often vulnerable. And, I’d suggest, offering a “solution” with no problem attached to it apart from that fear. Not to mention the issues around privacy, the fact that however many “it’s safe and encrypted” services get hacked, or sold on when the IPO comes around.
2) As another commenter says below - this is symptomatic of a type of parenting which ultimately creates fearful, anxious, badly adjusted children. Study after study shows that “free roam” children end up as better balanced humans. I’m not saying that watching and monitoring and stat-ifying your newborn is immediately going to make them anxious, but it says something about parenting which to me is unhealthy, obsessive, and ultimately not about freedom for the most important beings here: the kids.
I’m extrapolating of course, but hopefully my drift is somewhat clear..
> Different folks parent differently, culture evolves.
You are framing it as if every change is in a positive direction, which it clearly isn't. Risking at sounding like an old man yelling at clouds, look at the kids these days. They are so dependent, and so sensitive to negative stimuli or emotions.
Parents need to realise that they need to grow adults, not perpetual kids.
> look at the kids these days. They are so dependent, and so sensitive to negative stimuli or emotions.
I think many of us have found people writing comments like this are not interacting with children very much. More just reading the takes of others who also don't interact with children.
And if this was a legitimate problem to address, you would not address it by taking away baby monitors.
The topic of the conversation widened a bit from baby monitors. I of course do not advocate that taking away baby monitors will fix our dilapidated societies.
And while it is certainly true that I don't interact with young children a lot, the case is different for older ones or "young adults".
This feels like they were using the wrong architecture from the start, and are now papering over that problem with additional layers of cache.
The only practical reason to put a video in S3 for an average of 2 seconds is to provide additional redundancy, and replacing that with a cache removes most of the redundancy.
Feels like if you uploaded these to an actual server, the server could process them on upload, and you could eliminate S3, the queue in SQS, and the lambdas all in one fell swoop...
Yes, it's simple, S3 is for storing objects, not for processing.
Don't know how they came up with such a bad and complicated cloud design for something that is straight forward.
It’s a pattern prominently featured in AWS docs… upload to S3, react to CloudEvent in SQS, download and process with Lambda, upload back to S3…
I'm curious how many engineers per year this costs to maintain
> I'm curious how many engineers per year this costs to maintain
The end of the article has this:
> Consider custom infrastructure when you have both: sufficient scale for meaningful cost savings, and specific constraints that enable a simple solution. The engineering effort to build and maintain your system must be less than the infrastructure costs it eliminates. In our case, specific requirements (ephemeral storage, loss tolerance, S3 fallback) let us build something simple enough that maintenance costs stay low. Without both factors, stick with managed services.
Seems they were well aware of the tradeoffs.
And I am curious how many engineer years it requires to port code to cloud services and deal with multiple issues you cannot even debug due to not having root privileges in the cloud.
Without cloud, saving a file is as simple as "with open(...) as f: f.write(data)" + adding a record to DB. And no weird network issues to debug.
> as simple as "with open(...) as f: f.write(data)"
Save where? With what redundancy? With what access policies? With what backup strategy? With what network topology? With what storage equipment and file system and HVAC system and...
Without on-prem, saving a file is as simple as s3.put_object() !
>> Without cloud, saving a file is as simple as "with open(...) as f: f.write(data)" + adding a record to DB.
> Save where? With what redundancy? With what access policies? With what backup strategy? With what network topology? With what storage equipment and file system and HVAC system and...
Most of these concerns can be addressed with ZFS[0] provided by FreeBSD systems hosted in triple-A data centers.
See also iSCSI[1].
0 - https://docs.freebsd.org/en/books/handbook/zfs/
1 - https://en.wikipedia.org/wiki/ISCSI
Except running ZFS on FreeBSD would certainly require dedicated devops person with very specific skillset that majority of people on market dont have.
With s3, you cannot use ls, grep and other tools.
> Save where? With what redundancy? With what access policies? With what backup strategy? With what network topology? With what storage equipment and file system and HVAC system and...
Wow that's a lot to learn before using s3... I wonder how much it costs in salaries.
> With what network topology?
You don't need to care about this when using SSDs/HDDs.
> With what access policies?
Whichever is defined in your code, no restrictions unlike in S3. No need to study complicated AWS documentation and navigate through multiple consoles (this also costs you salaries by the way). No risk of leaking files due to misconfigured cloud services.
> With what backup strategy?
Automatically backed up with rest of your server data, no need to spend time on this.
> You don't need to care about this when using SSDs/HDDs.
You do need to care when you move beyond a single server in a closet that runs your database, webserver and storage.
> No risk of leaking files due to misconfigured cloud services.
One misconfigured .htaccess file for example, could result in leaking files.
> One misconfigured .htaccess
First, I hope nobody is using Apache anymore, second, you typically store files outside of web directory.
Why nobody should use Apache? I rediscovered it to be great in many use cases. And there's llms to help with the config syntax.
Performance not great compared to nginx.
>> No risk of leaking files due to misconfigured cloud services.
> One misconfigured .htaccess file for example, could result in leaking files.
I don't think you are making a compelling case here, since both scenarios result in an undesirable exposure. Unless your point is both cloud services and local file systems can be equally exploited?
With bare-metal machines you can go very far before needing to scale beyond one machine.
It sounds like you’re not at the scale where cloud storage is obviously useful. By the time you definitely need S3/GCS you have problems making sure files are accessible everywhere. “Grep” is a ludicrous proposition against large blob stores
I mean you can easily mount the S3 bucket to the local filesystem (e.g. using s3fs-fuse) and then use standard command line tools such as ls and grep.
I inherited an S3 bucket where hundreds of thousands of files were written to the bucket root. Every filename was just a uuid. ls might work after waiting to page though to get every file. To grep you would need to download 5 TB.
It's probably going to be dog slow. I dealt with HDDs where just iterating through all files and directories takes hours, and network storage is going to be even slower at this scale.
I don't think any of those mattered for their use case. That's why they didn't actually need S3.
You can't ever definitively answer most of those questions on someone else's cloud. You just take Amazons word for whatever number of nines they claim it has.
Not needing to ask the questions is the selling point.
Bro were you off grid last week. Your questions equally apply to AWS, you just magically handwave away all those questions as if AWS/GCP/Azure outages aren’t a thing.
Until it goes down because because aws STILL hasn't made themselves completely multi-region or can't figure our their DNS.
A lot of reductive anti-cloud stuff gets posted here, but this might be the granddaddy of them all.
Ah that is where logging and traceability comes in! But not to worry, the cloud has excellent tools for that! The fact that logging and tracing will become half your cloud cost, oh well let's just sweep that under the rug.
> Without cloud, saving a file is as simple as "with open(...) as f: f.write(data)" + adding a record to DB. And no weird network issues to debug.
There may be some additional features that S3 has over a direct filesystem write to a SSD in your closet. The people paying for cloud spend are paying for those features.
Variation on an old classic.
Question: How do you save a small fortune in cloud savings?
Answer: First start with a large fortune.
A small fraction of 1, probably? It sounds like a fairly simple service that shouldn't require much ongoing development
Especially if you have access to LLMs.
You're going to run a production system with a bus number of 1?
I think you mean a small fraction of 3 engineers. And small fractions aren't that small.
So far I have seen a lot more production systems with a bus factor of zero than production systems with a bus factor greater one.
The cost being a fraction of 1 does not imply it's one person. 3 people each spending 2 weeks a year on the service is still a fraction of 1.
It is three opportunity costs. No free lunches.
Nobody implied it was free. Yes there are opportunity costs, and they add up to less than one sysadmin of opportunity.
Yes, that was my thought as well. Breakeven might be like 1 (give or take 2x)?
Anything worth doing needs three people. Even if they also are used for other things.
What I notice, that large companies use their own private cloud and datacenters. At their scale, it is cheaper to have their own storage. As a side business, they also sell cloud services themselves. And small companies probably don't have that much data to justify paying for a cloud instead of buying several SSDs/HDDs or creating SMB share on their Windows server.
So, you want a place to store many files in a short period of time and when there's a new file, somebody must be notified?
Have you ever thought of using a postgresql db (also on aws) to store those files and use CDC to publish messages about those files to a kafka topic? In your original way, we need 3 aws services: s3, lambda and sqs. With this way, we need 2: postgresql and kafka. I'm not sure how well this method works though :-)
I have always understood S3 is just HDFS with some extra features? So, if you were going to roll your own S3, then you’d stand up an HDFS cluster.
because "How we stopped putting your kids in S3 buckets"
just sounded less attractive
They can't say that, as they did not stop. They made a cache in front of it.
I’m mostly just impressed that some janky baby monitor has racked up server fees on this scale. Amazing example of absolutely horrible engineering.
Also, just take an old phone from your drawer full of old phones, slap some free camera app on it, zip tie a car phone mount to the crib, and boom you have a free baby monitor.
If you don’t have fifty to a hundred dodgy PoE cameras from Alibaba tied to the crib do you even really love the baby?
In a capitalist society it is intuitive to outsource every ounce of effort to somebody else.
S3 certainly saves a lot of hassle, but in certain use cases, it really is prohibitively expensive. Has anyone tried self-hosted alternatives like MinIO or SeaweedFS? Or taken even more radical approaches? How do you balance between stability, maintenance overhead, and cost savings?
MinIO has moved away from having a free community fork, and I think it's base cost is close to $100k a year. I've been using Garage and been happy, but as a single dev and orders of magnitude smaller than the OP, so there are certainly edge cases I'm missing to compare the two.
I'm a fellow new Garage user. I have had a great time so far - but I also don't need much. My use case is to share data analysis results with a small team. I wanted something simple to manage that can provide an s3 like interface to work with off the shelf data analysis tools.
Some stuff like this also exists: https://www.dell.com/en-in/shop/storage-servers-and-networki...
We could just use something like that
Or there is that other Object storage solution called R1 from Cloudflare.
* R2
I'd it's processed in 2 seconds, why not just process it immediately in memory?
Because they are serverless, so there's currently no memory for it to be processed in at the point of upload
Maybe there are too many requests, so they have to offload the videos to s3.
Couldn’t they have used S3 express one zone?
Video processing is one of those things that need caution when doing serverlessly. This solution makes sense, especially because S3s durability guarantees aren't needed.
Ah, these modern babies. They can't even sleep without being spied and recorded 24/7.
I'm glad both me and my kind grew up in different times.
Today's kids will never in their lives know what freedom is, and we are guilty we made such dystopian societies a reality.
If anyone here uses the Nanit app in the background of their phones, it absolutely destroys battery life.
I got a new phone because I thought my battery was cooked, but turns out it was just the app.
Their architecture is internet bandwidth heavy and storage heavy; these are some of the most expensive things in AWS. You probably want to use a different provider for those things.
> It turns out that when AWS says an instance can do “Up to 12.5 Gbps”, that’s burstable networking backed by credits; when you’re below the baseline, you accrue credits and can burst for short periods.
Yes, AWS has a burst rating and a sustained/baseline rating for both EBS types as well as instance types. Use https://instances.vantage.sh/ (and make sure you choose specific columns) to compare specific criteria and then export as a CSV to find the lowest price that matches your performance/feature/platform criteria. Design to the baseline if you need guaranteed performance. Do sustained performance testing.
> When we Terminated connections idle >10 minutes, memory dropped by ~1 GB immediately; confirming the leak was from dangling sockets. Fix: make sockets short-lived and enforce time limits.
We used to do that with Apache 20 years ago. Config option forces a forked subchild to exit after N requests to avoid the inevitable memory leaks. AKA the Windows 95 garbage collector (a reboot a day keeps the slowness at bay).
FWIW, if the business feasibility of your architecture depends on custom stuff, performance enhancements, etc, you will find that you eventually have harder and harder problems to solve to keep your business functioning. It's more reliable to waste money on a solution that is brainless, than invest human ingenuity/technical mastery in a solution that is frugal.
They're very ingest heavy compared to how much of it is actually streamed out and to a very small/local audience so probably don't even need a cdn. And ingest on aws is free.
On the other hand, S3 is kind of ridiculously expensive compared to even more expensive on-prem options like a PureStorage SSDs array. With spindles on Ceph you can probably get quite a bit lower than AWS's 2c/Gig/mo. Or you can just use R2 with colocated servers for ingest.
unnecessary cloud subscription service abjures unnecessary cloud subscription service
they don't seem to have factored in the cost of doing this, so not sure what their actual saving was although it was probably substantial.
Cloudflare R2 solves this
Who is “The South Korean Government”?
It's the government who lost 850TB of citizen data with no backups[0] Because Cloud bad.
[0] https://www.techradar.com/pro/security/the-south-korean-gove...
Storing the data in a foreign cloud would allow foreign nation to play funny tricks on the country. What they need is not the cloud but sane backup system.
Isolated partitions exist.
ap-northeast-2 is literally in Seoul
CLOUD Act.
because they didnt have a decent backup.
I mean the "S3" could be replaced with object storage. I guess thats the technical term anyway. Having said that just goes to show how cheap S3 is, if after all of this, the savings are just $500k. Definitely money saved but not a lot.