It's driven by aggregated user complaints, so it's pretty high-quality. Comment counts are a proxy for "how bad was the outage" + "how long was the outage thread topical for".
GitHub CI is a very weird product. My team pays a decent bit of money for it, and we are nowhere near hitting the minutes limit, but at the same time we easily run into the storage limits, and the runner is incredibly slow?
On top of that the tooling is just... not good. The more I use it, the more I get annoyed at weird UX decisions, strange missing features, and the whole thing just being a giant bowl of spaghetti in general. You can easily spend days trying to make your workflows not suck just to realize halfway through that you could've been done already if you had just handwritten the damn thing in Bash.
We now live in the insane world where I am glad our tickets are still in the nightmare called Jira, because every day I get closer to asking management to switch to a forge which is actually functional - and with only CI and Git the transition would not be very difficult...
One of the things I didn't get to do at my last place before leaving was that I was putting together a proposal to beef up our CI infrastructure to be more cores per agent. I was trying to figure out the math to prove we would save money by doing so.
I had a batch process I became responsible for that ran on the same VM as an important production service and every once in a while it became a noisy neighbor. It ended up getting moved onto a much smaller cluster for the compute part of the operation after I did some pretty aggressive traffic shaping work to get the cost down by more than a factor of 10.
At one point I got the code into a shape where I thought I could just run it directly on the CI machine, and the lower CPU availability would be offset by the higher IPC overhead and cache misses. Problem was that it was still about 2.3x slower running on 1/3 as many cores. But those VMs idled for hours at a time while waiting for a 10 minute job to be started.
Near as I could figure, if we doubled the size of the instances the build agents ran on, I could shut down that cluster and just about offset the cost of the more expensive build agents exactly, and speed up quite a lot of builds in the process (many build steps do not parallelize but some do).
If one other team had a batch process they could just run directly on the CI machines, that would be a net reduction in operating costs. We had quite a few teams doing batch processing. CI servers have a lot of tricks you can use to queue up tasks rather than running them concurrently.
I think it's brilliant. I'm now paying $30 per month for Github actions as a duo shop... it's freemium working the way it should. Give them the first hit and all.
And yes, I need to optimize (and am), but they're already getting hundreds from little me. I even started going towards the CI doing browser walks, code tests, AI rework, etc. It saved me from having another dev machine.
Great stuff, powerful as all heck, and once you see the power it was—for me—impossible NOT to think of 6 other uses and blow through the free CI tier.
The thing is, once you start paying for it, then a service like blacksmith is so attractive and easy to replace gh, that it seems a no brainer. We swapped like 30-40 gh actions from gh runners to Blacksmith runners and cut our ci times by about 4x on almost all of our workflows. And all you do is auth their gh app and update an image tag.
Especially if you can take advantage of the high vcpu runners, then you can get some really fast pipelines going (not affiliated w/ them, just an enjoyer).
I started on free, then upgraded to a 'pro' subscription. Now I'm paying the pro subscription + ~ $300 / mo in additional runner costs. Now looking at starting to self-host some runners to offset that growing cost.
I've started hosting most of my private projects on a self hosted gitea, but one roadblock I've found is oidc auth - from GitHub actions I can publish to the Google play store, or npm, without needing to manage any static credentials. I know I'll need to manage some kind of secrets/identity when self hosting, though I'm still looking for nicer solutions than a bunch of static secrets for individual services.
It had huge market traction but it all evaporated when they changed the offering for FOSS. I wonder what would happen if Github announced tomorrow that all actions require a subscription, where would users go?
The whole actions/CI thing is really nice but there's no good reason the computationally heavy stuff should be running on Github's (Microsoft's) servers. They can run the control plane cheaply and make the user run an agent somewhere to act as the build box runner thing. The user experience would be identical, but they'd outsource the vast majority of the compute costs to the user[1]. It's shocking they didn't do something like that model in the first place, tbh. Moviepass for CI.
[1] It can also have supply chain security and reliability benefits too. You're not actually building deployment artifacts in your actions runners, right? Right? RIGHT?
Yes, that's what I meant. A lot of folks just build stuff in a Microsoft hosted actions runner which can talk to the entire Internet and pull deps straight from PyPI or whatever. Then those artifacts show up on the Releases page and go into production. Whee.
Not even that heavy handed. They just need to _severely_ rate limit the top 0.1% of free users that are raping their hardware. I bet they are whales in usage terms and the root cause of many heartache.
I've heard this claimed dozens of times. I wonder if anyone at Github can confirm this theory and maybe give insight into why they haven't just done this?
I wanted to set up a Cloudflare static page and all the tooling is completely designed around Github. If you want any other service, you have to roll it yourself.
Cloudflare has official support for Cursor Origin. I migrated all of my repos from GitHub to Cursor Origin and am still able to easily use Cloudflare Workers and Pages with all of its features, including preview builds from PRs in Origin.
Interesting! I should investigate that. I was following their docs and GitHub was 100% the happy path, and I was actually hoping to avoid it (downtime woes).
Yeah, the GitHub integration is definitely "click button, sign in to GitHub, automation sets everything up" which is simple and hands-off.
But, the wrangler CLI isn't that much of a step up. Biggest items are installing the CLI and having the CloudFlare API token available in a secure manner.
Isn't that why most people are on GitHub in the first place ? Which then, Windows style eventually turns into people wanting to use GitHub also for company work, where the real money comes from for GitHub.
Without free CI in GitHub, you might as well say:
"they can just shut the whole thing down and this reliability issue will be solved immediately"
Maybe it is, but I know that lots of us used GitHub way before Actions launched. I think even if Actions wasn't free, GitHub would still have a massive advantage because it is the most well known place to host code, especially by less technical users (which is where I see a lot of finger pointing going with the increase in GitHub usage).
I would not be surprised if a serious amount of this usage is not user-intended and instead the LLM just inserts some sort of CI job off the bat and people roll with it.
Yes, we will be offering official CI (though nowhere near as free as GitHub). We will also support 3rd-party CIs (like https://buildkite.com, https://depot.com) & self-hosted runners (similar to GitHub)
It bugs me how there can be an acknowledged live incident happening today with Actions, and yet down the page where it shows last 30d uptime for actions, today's little rectangle is a chipper green.
Easy: it isn't down, it is just "experiencing degraded performance" which impacts "a limited set of customers". It only counts as downtime when nobody can access any part of it!
I've been saying this on comments for ever - it sucks as a paying customer to have a company that could care less. MS is the worst here. The rest are pretty bad too. Satya just cares about sound bites and stock bumps from layoffs, little intelligent thought from him now unfortunately (he used to be decent, now useless)
Ever since I switched to Forgejo (could do Gitea or other self-hosted too), my outages have gone down to basically zero. It’s only down when I do maintenance but since I’m not pushing code at the same time it doesn’t matter
It's pretty easy to see where that ends. The headline writes itself: "Github engineers arrested for betting on downtime on prediction markets and intentionally causing outages"
Nice graphic on that page of a bunch of cartoon workers tending to what looks like a Rube Goldberg contraption. It's all just lighthearted fun and games.
We use a self hosted gitlab-ce instance which has pretty high up time running on a 10 year old server (and its backup)
Takes a few minutes a month to update and babysit. We only basically use the CI and the repository functionality though so gitlab has been "feature complete" for us for a while. Probably I we could swap to something like gitea or forgeo but the gitlab CI seems like a nice balance between powerful when it needs to be and simple otherwise for our use case so there is nothing driving us to swap.
The shape of this HN duplicate detector actually works as a GitHub uptime chart:
https://news.ycombinator.com/from?site=githubstatus.com
It's driven by aggregated user complaints, so it's pretty high-quality. Comment counts are a proxy for "how bad was the outage" + "how long was the outage thread topical for".
edit to add: Here's a scatterplot of the same,
https://files.catbox.moe/hhfh4l.png
Really smart insight lmao
lol
I think all they have to do is stop giving free CI to everyone and this reliability issue will be solved immediately.
Disclaimer: I am building a code forge myself at https://juju.bi
> I am building a code forge myself
Isn't everyone else doing that too?
* https://cursor.com/docs/origin
* https://tangled.org/blog/forge/
* https://radicle.dev/
* https://entire.com/blog/the-git-forge-for-agent-human-collab...
The space is getting a little cramped, to say the least.
I think they all have different philosophies in approaching the problem.
* https://juju.bi - Local-first architecture
* https://cursor.com/docs/origin - Scalable VCS backend (this is actually very easy to implement by others)
* https://tangled.org, https://radicle.dev - Distributed forges
* https://entire.com - Very agent-centric
free CI on github is so ridiculously overpowered it's hard to believe they still provide it
GitHub CI is a very weird product. My team pays a decent bit of money for it, and we are nowhere near hitting the minutes limit, but at the same time we easily run into the storage limits, and the runner is incredibly slow?
On top of that the tooling is just... not good. The more I use it, the more I get annoyed at weird UX decisions, strange missing features, and the whole thing just being a giant bowl of spaghetti in general. You can easily spend days trying to make your workflows not suck just to realize halfway through that you could've been done already if you had just handwritten the damn thing in Bash.
We now live in the insane world where I am glad our tickets are still in the nightmare called Jira, because every day I get closer to asking management to switch to a forge which is actually functional - and with only CI and Git the transition would not be very difficult...
One of the things I didn't get to do at my last place before leaving was that I was putting together a proposal to beef up our CI infrastructure to be more cores per agent. I was trying to figure out the math to prove we would save money by doing so.
I had a batch process I became responsible for that ran on the same VM as an important production service and every once in a while it became a noisy neighbor. It ended up getting moved onto a much smaller cluster for the compute part of the operation after I did some pretty aggressive traffic shaping work to get the cost down by more than a factor of 10.
At one point I got the code into a shape where I thought I could just run it directly on the CI machine, and the lower CPU availability would be offset by the higher IPC overhead and cache misses. Problem was that it was still about 2.3x slower running on 1/3 as many cores. But those VMs idled for hours at a time while waiting for a 10 minute job to be started.
Near as I could figure, if we doubled the size of the instances the build agents ran on, I could shut down that cluster and just about offset the cost of the more expensive build agents exactly, and speed up quite a lot of builds in the process (many build steps do not parallelize but some do).
If one other team had a batch process they could just run directly on the CI machines, that would be a net reduction in operating costs. We had quite a few teams doing batch processing. CI servers have a lot of tricks you can use to queue up tasks rather than running them concurrently.
I think it's brilliant. I'm now paying $30 per month for Github actions as a duo shop... it's freemium working the way it should. Give them the first hit and all.
And yes, I need to optimize (and am), but they're already getting hundreds from little me. I even started going towards the CI doing browser walks, code tests, AI rework, etc. It saved me from having another dev machine.
Great stuff, powerful as all heck, and once you see the power it was—for me—impossible NOT to think of 6 other uses and blow through the free CI tier.
The thing is, once you start paying for it, then a service like blacksmith is so attractive and easy to replace gh, that it seems a no brainer. We swapped like 30-40 gh actions from gh runners to Blacksmith runners and cut our ci times by about 4x on almost all of our workflows. And all you do is auth their gh app and update an image tag.
Especially if you can take advantage of the high vcpu runners, then you can get some really fast pipelines going (not affiliated w/ them, just an enjoyer).
I started on free, then upgraded to a 'pro' subscription. Now I'm paying the pro subscription + ~ $300 / mo in additional runner costs. Now looking at starting to self-host some runners to offset that growing cost.
It’s super easy to self host ci, I hosted one with a hetzner VM (pick one in the cheapest region, it doesn’t matter much where CI lives
I've started hosting most of my private projects on a self hosted gitea, but one roadblock I've found is oidc auth - from GitHub actions I can publish to the Google play store, or npm, without needing to manage any static credentials. I know I'll need to manage some kind of secrets/identity when self hosting, though I'm still looking for nicer solutions than a bunch of static secrets for individual services.
I still remember when TravisCI giving free CI to FOSS projects was revolutionary
It had huge market traction but it all evaporated when they changed the offering for FOSS. I wonder what would happen if Github announced tomorrow that all actions require a subscription, where would users go?
The whole actions/CI thing is really nice but there's no good reason the computationally heavy stuff should be running on Github's (Microsoft's) servers. They can run the control plane cheaply and make the user run an agent somewhere to act as the build box runner thing. The user experience would be identical, but they'd outsource the vast majority of the compute costs to the user[1]. It's shocking they didn't do something like that model in the first place, tbh. Moviepass for CI.
[1] It can also have supply chain security and reliability benefits too. You're not actually building deployment artifacts in your actions runners, right? Right? RIGHT?
In my self-hosted ones. Where are you building them??
Yes, that's what I meant. A lot of folks just build stuff in a Microsoft hosted actions runner which can talk to the entire Internet and pull deps straight from PyPI or whatever. Then those artifacts show up on the Releases page and go into production. Whee.
Not even that heavy handed. They just need to _severely_ rate limit the top 0.1% of free users that are raping their hardware. I bet they are whales in usage terms and the root cause of many heartache.
I've heard this claimed dozens of times. I wonder if anyone at Github can confirm this theory and maybe give insight into why they haven't just done this?
Im guessing the lock-in is strong enough that they can eat the reputational hit while getting meaningful training data by subsidizing the free users.
Github has a decade of positive reputation to deal out, I don't see Grok or OpenAI jumping in and competing with that.
I wanted to set up a Cloudflare static page and all the tooling is completely designed around Github. If you want any other service, you have to roll it yourself.
Cloudflare has official support for Cursor Origin. I migrated all of my repos from GitHub to Cursor Origin and am still able to easily use Cloudflare Workers and Pages with all of its features, including preview builds from PRs in Origin.
The official Cloudflare CLI can ship static site content with one command: "wrangler pages deploy"
Easy enough to run from another CI or automated system.
Interesting! I should investigate that. I was following their docs and GitHub was 100% the happy path, and I was actually hoping to avoid it (downtime woes).
Yeah, the GitHub integration is definitely "click button, sign in to GitHub, automation sets everything up" which is simple and hands-off.
But, the wrangler CLI isn't that much of a step up. Biggest items are installing the CLI and having the CloudFlare API token available in a secure manner.
I suspect the 1/4 (?) ownership by Microsoft plays a factor into OpenAI.
Otherwise I reckon they’d have cloned GitHub already; possibly with a “Mirror to CodexHub? Free usage reset”.
Isn't that why most people are on GitHub in the first place ? Which then, Windows style eventually turns into people wanting to use GitHub also for company work, where the real money comes from for GitHub.
Without free CI in GitHub, you might as well say:
"they can just shut the whole thing down and this reliability issue will be solved immediately"
Maybe it is, but I know that lots of us used GitHub way before Actions launched. I think even if Actions wasn't free, GitHub would still have a massive advantage because it is the most well known place to host code, especially by less technical users (which is where I see a lot of finger pointing going with the increase in GitHub usage).
I would not be surprised if a serious amount of this usage is not user-intended and instead the LLM just inserts some sort of CI job off the bat and people roll with it.
Do you have a CI story in jujubi? Already it has a nice set of features.
Yes, we will be offering official CI (though nowhere near as free as GitHub). We will also support 3rd-party CIs (like https://buildkite.com, https://depot.com) & self-hosted runners (similar to GitHub)
Hm looks interesting, I joined the wait list just now. :)
It bugs me how there can be an acknowledged live incident happening today with Actions, and yet down the page where it shows last 30d uptime for actions, today's little rectangle is a chipper green.
Easy: it isn't down, it is just "experiencing degraded performance" which impacts "a limited set of customers". It only counts as downtime when nobody can access any part of it!
There were several Actions outages in the past two weeks and none of them seem to impact the color of the rectangles.
A completely different, non GH controlled status page had to be made to accurately report status for them.
It's hard coded probably.
If you hover over the green bar for today, the tooltip knows there's an incident.
if day.endswith('y'): return GREEN
Lesson for every infra provider: physically separate your free tier infra from your paid tier infra.
I've been saying this on comments for ever - it sucks as a paying customer to have a company that could care less. MS is the worst here. The rest are pretty bad too. Satya just cares about sound bites and stock bumps from layoffs, little intelligent thought from him now unfortunately (he used to be decent, now useless)
Self-hosted GitHub Actions on AWS spot are still working.
After many outages we had to build our own free runner https://github.com/supercorp-ai/superci
This works as long as the schedulers don't go down (which they have, repeatedly in past incidents)
Ever since I switched to Forgejo (could do Gitea or other self-hosted too), my outages have gone down to basically zero. It’s only down when I do maintenance but since I’m not pushing code at the same time it doesn’t matter
Has anyone tried https://tangled.org/ for simple projects on their home server? Is it easy enough?
I wonder if I can hook failed git push into betting on github's status on polymarket.
It's pretty easy to see where that ends. The headline writes itself: "Github engineers arrested for betting on downtime on prediction markets and intentionally causing outages"
No way, is it a day ending in "y" again?!
I'm sure there is a good meme to use for such a frequently occurring problem, but it would be too over used by this point.
My Actions are stuck waiting :-/
same here
like clockwork! right now on the frontpage:
The federated code forge (https://news.ycombinator.com/item?id=50305573)
No way I don’t believe it
I was confused why the github file browser was falling over.
Giving Github more attention
It's getting to be hilarious at this point, honestly.
Another day another GHA incident
[flagged]
[flagged]
[dead]
[dead]
Nice graphic on that page of a bunch of cartoon workers tending to what looks like a Rube Goldberg contraption. It's all just lighthearted fun and games.
Last time these intermittent incidents happened was days ago and for this exact same incident was 11 days ago. [0]
After everything announced in GitHub Universe and with that to improve the reliability of GitHub overall, yet we have another incident afterwards.
It doesn't matter what GitHub does, only you can decide for yourself to self-host rather than going all in on GitHub and it continues to go down.
Predicted this 6 years ago and self hosting has aged perfectly well. [1]
[0] https://news.ycombinator.com/item?id=50154406
[1] https://news.ycombinator.com/item?id=22868406
We use a self hosted gitlab-ce instance which has pretty high up time running on a 10 year old server (and its backup)
Takes a few minutes a month to update and babysit. We only basically use the CI and the repository functionality though so gitlab has been "feature complete" for us for a while. Probably I we could swap to something like gitea or forgeo but the gitlab CI seems like a nice balance between powerful when it needs to be and simple otherwise for our use case so there is nothing driving us to swap.
Yep. Ironically this downtime is currently delaying our ongoing migration to self-hosted forgejo...