> Errors in those services triggered a client-side retry loop that increased traffic during recovery
Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.
> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.
A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded).
If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go great. But there are lots of situations where one or both of those breaks down.
Maybe the retry logic was vibecoded instead of using an existing hardened library. After all, according to Twitter, nobody is looking at the code anymore.
Except the other side of this is interrupting a service which would otherwise have succeeded: there's a lot of unattended or minimally attended processes where an interruption is just asking the user to do the only thing they were going to do anyway - retry it.
In GitHub's case this is especially relevant - the only reason to throw an error message at the user is the hope they - the human - give up and walk away (or you break all the CI/CD builds and the time it takes humans to hit "retry" gives you some breathing room).
Are retries bad?
These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic.
I feel like I'm mildly stupid in trying to out retries as heresy but I'm not sure.
Commits are not expensive, pushes are. You can do any number of commits before you do one push, unless you are editing online, in which case every act is it's own commit & push.
You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.
>You can rig up a local ide to pathologically commit+push per save
The dev system we use for a 3rd party hosting provider (a big one) requires a commit and push for every file save while we're developing. I created a build system for this that copies the whole repo to a temp folder. As we save changes to files in the main repo folder, the build system watches for changes and copies the changed file to the temp folder, then does a commit on the temp folder and pushes to a an intermediary repo in github which then triggers an action that causes the 3rd party system to update from the intermediary repo. This way we don't pollute our main source repo with a commit every time we save an update to a source file.
It's not my favorite way to develop but it's caused us no real problems except when github goes down.
Well either they can handle this load that Microsoft can't, or they can't. If Microsoft are going to continue to be unreliable in the absence of the rate limit then:
If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway.
If alternatives aren't able to manage, then user's aren't going to jump since those services won't actually provide more usage.
I still remember when they decided to limit the number of private repos you could have as a free user. Kind of silly to me at the time, and even more so now!
Which will just increase the cries of "enshittification" and hasten the mass migration to the next free platform that surely, this time, won't ever go down.
We're small enough that we've been hosting our git infra for about a year now, I wonder how many other companies figured out they could make the trade. I've had a Github since a couple years after they started and I think they are going to become a Stack Overflow, albeit slower with MS at the helm. If Github is going to be 99% slop it's going to be really hard to use as a fun tool to show what you can do, what you've worked on, side projects, etc. I took github off my resume and I'm probably not going to relaunch my weblog if I end up job hunting, too much low-effort crap and people basically copying what a lot of us had been doing manually for years to really feel like it's anything other than a negative signal.
Not sure to be honest, from a machine perspective 2X should never be a big deal, unless 1.4 was the threshold or sweet state and no one thought too much about scale and architecture beyond that
If you’re the size of GitHub and you’ve been running your infra for years with very little variation in traffic patterns you have a strong incentive to optimise costs for that existing behaviour.
I had an interview with someone who has refused to use AI, and he has been unemployed for 2 years since graduating.
On the bright side, if my 6 year old can make 3D video games, I expect seasoned programmers to be able to pick it up quickly. I think ego is the hardest thing to break.
Not that impressive when you realize it's mostly due to AI slop
edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.
Agree - this reminds me very much of the old joke of two economists increasing GDP by taking turns giving the same 200usd back and forth for having each other eat shit.
Useful / impressive for whom is the question.
Not for us!
We pay for Github enterprise, and because GH can't be bothered to separate service tiers for sloplords and actual paying customers we get garbage level performance. They could of course always implement usage limits, but the goal is not to earn money, or provide a good service, the goal is to maximize AI users. Would be very awkward at the next executive golf meetup if you couldn't point to increased AI adoption.
In short: This is why monopoly laws matter. Once a company becomes too large, normal business rationales cease to be the motivation for their actions, and GH can go along with the pied piper of AI psychotic C-suite officers like MS is doing instead.
Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).
How is it impressive if we all know it's autogenerated? There's no more people there than there were before. Heck, at ~2x growth that's possibly a decrease in real humans there since bots generate loads of them per person
Yeah, seems like AI slop is going to kill GitHub's free tier. I just don't see how the economics of having to host this much slop and provide service to slopcoders is going to convert into dollars for them otherwise. None of the humans involved are going to end up in big enterprises. It's all cost, with no pathway to revenue.
Is it impressive? All it's doing is decaying the services. 15 years ago never have imagined I would go to the lengths to host a github alternative on a VPS but after doing just this (also being the last one in the my professional group to do so), GitHub is giving a master class in destroying their reputation in pursuit of advocating for hostile entities.
Not all growth is good, especially growth that is actively hurting the company.
There is an equilibrium in both nature and software. Purposely designing systems that mimic the effects of cancer is going to benefit who exactly?
This growth is surely good for github. If another company becomes the "github for AI agents", they'll lose not just their business for AI but also human coders. (Sure, maybe there will be a human coder only github, but it will be quite small.)
When a service that was already the primary git hosting provider for most of the world for 20 years grows at that rate its not mundane and its not comparable to any example.
Eh, I would be more empathic in this situation[0].
Github isn’t small startup, where other 10x threshold is as cheap as buy bigger box in your IaaS.
When you are already biggest player in the ecosystem and you suddenly get 10x persisted traffic, with at least 30x+ forecast “soon” - I am not surprised they have issues.
Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI.
I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.
Conversely, what suggests GitHub has a huge operating cost?
Running a GitHub clone at their same scale as a customer on cloud pricing would likely be insane. But y’all know infra is actually quite cheap when you run it yourself right?
It’s usually the case with these M&A deals that the profit just never quite makes sense to justify the purchase price, unless you can truly scale up the user base or revenue model. GitHub was already so mature as a solution when they bought it, I don’t know that they could have added that type of value just by slapping a Microsoft logo in the footer.
operating at a loss and non-operational because of outages are very different. If they can't maintain service levels nobody - AI super user or quant, old-fashioned human - will be happy.
Possibly dumb/silly question.... Are there any sorts of reverse proxies out there that provide their own layer of jittered/exponential backoff based on patterns? (i.e. requesting IP, cookie, etc.)
I suppose the main reason I think it might be a bad idea, is that it would add complexity to the reverse proxy (i.e. now it's having to track whatever thing is being used and that complexity itself becomes a potential failure point...)
(To be clear, the clients should have their own backoff procedures, but I'm thinking about cases involving naughty clients, which are sometimes a harder problem to correct for...)
I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference.
Do you know of a documented instance where it would have helped?
It’s not hard to get started, it’s a case of adding small amounts of randomness.
If you have, say, a long poll then kick off all users due to a deploy or error (or a broadcast message) then you can have a situation where you’ve got a huge clustering of connections at 1 minute, which spreads very slowly out as real life issues give you jitter for free. You can avoid this or at least return to normal much quicker by adding some jitter.
It might happen if all your users back off at the same rate too, if the clustering causes a bunch of errors. Error -> lots reconnect 1 minute after -> fail -> lots reconnect 2, 4…
More likely to occur in cases where there’s a way you can have people all connecting at the same time - synchronisation to a real world event is one case and then connecting again at the same time after.
This doesn't answer your question, but I faced an issue where an application had to retry if things failed and I ended up with the "thundering herd" problem and introduced jitter without knowing it was a relatively standard practice.
I felt dirty implementing such a solution (introducing randomness for the sake of randomness is off-putting), but it worked. It wasn't until a while later that I even heard the term "jitter" in this context and realized this was a pretty decent solution for this kind of problem.
Regardless, if you're going to introduce something where adding jitter is appropriate, I'd just just add jitter. It's not premature optimization; it's an essential part of that kind of functionality.
Downstream database of our edge serverless platform went down. A tonne of requests failed all at once. Every service in the microservice request path, and the client, had their own retry policy.
Clients all retried at the same time. Retries amplified in our microservice graph (1 request at the front door ended up with like 10s of retries internally as each downstream microservice along the path retried requests). Request queues backed up and couldn’t drain fast enough. Clients all timed out at roughly the same time. All waited the same time. All retried again at the same time.
It was a pulsing thundering herd of many hundreds of thousands of requests at the front door that was amplified by internal retries.
Had to tune up load shedding to 100% after the database outage was mitigated until the backend recovered then tune it down in increments to restore service.
Added jitter to clients and turned off retries on the serverless platform.
If you’re talking about internet clients, I think the real world provides sufficient jitter. If you’re talking about a fleet of clients on your 10gbps network, jitter might be useful.
I've had to simulate jitter recently to reproduce a memory spike in a go app using traffic control (tc). have you observed how your app works under jittery network?
Great read - I'm glad they realize there's work ahead but what I'm missing is:
* Paid customers: we know you pay us often a ton of money, and we burn your month on actions during these outages - we'll refund you for the days we spent your money and gave you no value.
* Paid customer: We know you put your trust in us, so we'll ensure we have a separate pool of capacity to ensure we can keep that trust.
* Paid customer: we'll proactively refund you when we miss our SLA.
What I read from this is:
* Scaling is hard, we don't have enough capacity
* We give away a shitton of compute for free
* I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.
Notice there's nothing about paid customers, I'll add in what they are missing:
Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.
This. I own a small company with 5 people. I pay Github $250/m. I'm sorry but the narrative of, "look at this burden we have, it's hard to take care of all of this code!" is pretty insulting when I'm paying $50 per person per month to host code and run CI pipelines. If they do not want my money, I'll find a company who does.
Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.
> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations
How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.
> this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started.
Well that's because in comparison to the absolute flood of traffic brought on by AI, they really haven't been operating on this scale before.
The trend line does not tell you what will actually happen at scale, even if you think you're perfectly prepared for the next 10% or 20% growth. As Mike Tyson put it, "everyone has a plan until they get punched in the face".
The problem is there are a class of problems that only appear after you go over the tip of what your system can handle, which are very difficult to predict or model.
Unfortunately time and time again, the overwhelming majority of people show that they would rather deal with an unreliable scummy company for "free" than pay $1/mo for a reliable service which treats them with respect :(
OK that "treats them with respect" is certainly an important aspect though. It does not seem like GH is especially great on that one, including listening to its users.
I said in another thread that they can't blame increased demand for these outages, but the demand growth is genuinely insane for a company already operating at huge scale.
I guess we'll have to wait and see if they deliver now, but it seems like they're taking it seriously at least.
"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."
Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.
Github is ripe for disruption and I hope it is disrupted soon.
"While accelerating our migration to Azure," meaning, they will only solve problems if it helps them also use Azure more.
It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage, unless they have no reporting or their tooling is completely incompetent. If things can fall apart so easily, throwing more capacity at the problem won't fix it.
You’re torturing your own logic to make Azure the villain here. And it also sounds like you lack experience with capacity exhaustion. Things fail slowly, then suddenly.
No, I am calling Github disingenuous liars with poor motives, offering an unbelivable explanation. Not sure where the torture is, except in their statement.
If it was failing slowly, why didn't they fix it then? Why didn't they notice?
Could it be because they fired all the people who know how to monitor and identify such problems? That's what SREs are for.
I don’t owe you anything, much less a separately sourced counterargument. You openly admit that your opinion is not based not on GitHub’s proffered statements but on your self-admitted assumption that GitHub is actively lying in an attempt to cover up an Azure-related root cause.
I’m firmly in the camp of “something stinks at GitHub” but
> It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage
In my experience, there are hard thresholds that get passed that expose hidden bottlenecks like this. A previous system I worked on we had absolutely loads of headroom by all of our measured metrics, but one day we filled a cache because the value hadn’t been tweaked in recent memory. Plenty of space on disk and in memory, but all of a sudden we went from a very high cache hit rate to a very low cache hit rate, and everything ground to a halt.
This absolutely can happen in large systems. If some part of the system is at capacity, then slightly increasing the load can cause it to fall behind and start accumulating a backlog.
These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.
Sure. But you might not even be realizing that something is just at the cusp if the load is spiky enough.
The art of large system design is to identify and avoid these kinds of chokepoints. And when something happens, propagate the "backpressure" up the stack to avoid queuing.
I'm betting on Tangled and Codeberg. Tangled has a better press and in general is a dark horse, Codeberg has the "brand" and some network effects from projects that moved to there. (famously, Zig.) I heard that Sourcehut is having a moment as well, and I love the idea of email-based workflow and not having to have an account to contribute to someone's project hosted there, but I'm not maintaining anything worthwhile paying the $4/mo sub.
that is more manageable but c'mon I can't even keep Google One 100GB up on a consistent basis, that's how poor I am. self-hosting would be a far better option because apparently I find enough people to provide free Hetzner VPSes and stuff as long as I can sell this as mutually beneficial.
for context, I would GLADLY move there my Neovim plugin. all it does is brings the current jj message into your editor and lets you integrate it with a status bar (or anything in nvim, really). that would be a decent measure against drive-by slop contributions, and I'd accept contribs over private github mirror from those who I know but can't bother setting up git mail
EDIT: TIL that one can host SourceHut themselves. discoverability may still be a problem (sr.ht just ranks higher in search engines) but 1) fixable with github mirror that points to sourcehut instance as a canonical development platform, 2) it's moderately easy to sync contributions between tangled and sourcehut, so tangled is also an option
EDIT 2: the email part would be PITA, so $4/mo is attractive on that background
Ok, but there's no universe where a major Microsoft-owned property is not being forced to run on Azure. Just like AWS pushing to get off Oracle back in the day. It would be career-destroying to suggest otherwise regardless of technical merit (and tbf, no infrastructure is bulletproof, unless you want to port GitHub to z/OS on mainframe)
If you're a big company, you can afford having one engineer spend one or two days per year to maintain your self-hosted GitLab or Forgejo. On top of better reliability than GitHub, you'll get the additional bonus that your source code won't accidentally leak through being in Copilot's training set.
If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.
Speaking from experience, it cost mW about a week or two per year to maintain GitLab for the startup I worked at.
My personal GitLab on the other hand really does take only a day or two per year.
That said, a week or two per year is just what it costs to maintain any one thing period. I spent about that much time maintaining PCs in the office, or my personal proxmox setup. It's not onerous at all.
GitLab is super bloated and a little sucky to admin, but it's not too bad all things considered. I'm admin in my new job's GitHub org and it sucks a whole lot more to maintain.
The issue with these systems is that they lack Github's sophistication for issue tracking, knowledge transfer, and automation. I think Gitlab is a mature product in its own space and unlikey to change, for instance, at this point.
Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but it's unattractive to me - as someone who agrees with most of their current politics - because the day they decide they don't like me, I'm screwed.
I never found GitHub's systems for issue tracking to be all that great. Cross-repository issues and development plans are hard to track within a git host. I've always used an external panning and issue tracking tool, mostly Linear, and it works really well. GitLab's Linear integration is excellent, FWIW.
I've actually worked with a couple of companies who do use GitHub for their code, and they all use Linear in addition to GitHub.
I understand the concern you're talking about wrt. Codeberg, but I wouldn't view it as a significantly bigger risk than anything else. Any platform can suddenly decide that your project is against ToS (GitHub will absolutely not accept just anyone's use of their platform either) and Codeberg introducing some rules recently doesn't, in my mind, drastically increase the risk of a dramatic ToS change in the future. But we all have to make our own risk evaluations and I won't judge yours. Luckily, moving between Git hosts isn't that difficult; setting up CI again and losing merge request history does suck but it's not the end of the world, unlike something like, say, losing your AWS/GCP/whatever account.
"sophistication" seems like a strange way to describe GitHub to me - I've found in every individual aspect (code browsing, issue tracking, code review, package management, etc), it's the worst out of all the systems I use regularly... But it's good _enough_ for most people, and it has all those features in one place, which is more convenient than wrangling 10-15 high quality but disconnected systems
Gitlab has the benefit of having very little traffic, both free and paid. Their limits are still way above the current usage so less likely to be an issue
Worth mentioning GitLab's paid enterprise offering are more expensive than GitHub's, on a per-seat basis.
Lots of companies moved because it was cheap, but it's not anymore. Ironic that companies might choose to migrate to them now for stability, rather than price.
Dude no one uses github tracking for anything serious, come on. The only thing github has over gitea/forgejo is discussions. That's the only real social "innovation" github has contributed to open source development and it's just a shitty tacked on forum.
Also for accuracy, Codeberg has a pro-human and anti-corporation stance. Both of which are definitely en vogue at the moment.
Much better than GitHub's pro slop sentiment, which is doing nothing but destroy their reputation.
> We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
A server box now has around 256 CPU cores. So that's about 12000 servers. If each one is $10k that's $120 million. Not a lot compared to Github's income.
$10,000 is only enough for CPU cost (2x AMD EPYC 9754). The rest of the servers (mainboard, RAM, storage, power supply, network card, rack, cooling, etc.) can easily triple or quadruple the cost.
Microsoft don't release the costs as you know, but
Compute and Storage for Free Tiers: Hosting code for over 150 million developers and processing over 2 billion GitHub Actions (CI/CD) workflows a month requires astronomical server power and data storage. The "Free" tier is a massive cost sink that Microsoft treats as a loss-leader marketing expense
Let me know when you understand how that's not free.
We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.
GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.
Forgejo is pretty neat. Their CI story is sad because it's based on act and it lacks features like GH Apps so service accounts are not possible (using a user account as service is barf).
I self-host it and its pretty easy to have uptime higher than github when you have 3 users total.
"Forgejo is a self-hosted lightweight software forge"
That says absolutely nothing. The "What is Forgejo?" question is unanswered and instead you get a lot of words about their values, their inclusivity, etc. And the next thing in the docs is how to install it. It's ridiculous.
> a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.
And another outage. [0] Looking forward to the subsequent post-mortem on that one.
You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.
6 years ahead [1] on not going all in an centralizing everything on GitHub.
Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.
This stuff is good! This is what a booming economy looks like. There are people out there competing with you for resources because they have cool ideas they want to implement.
Or at least they asked the AI to come up with cool ideas, which is even more interesting. It's exciting watching the world transition away from humanity being in the driver's seat!
How would any of what you're saying help with this?
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.
load balancer failure? rate limit woudl address
concurrency limits? throttle or queue up messages. auto-scale failed cause was misconfiguration policy, which i admit cannot be handled by my suggestions.
The cascade? it's downstream service degradation, which I mentione should have had been prevented with queues. One of the jobs that queues/kafka solve is to prevent these downstream outages.
If your LB is down, you're just kind of screwed. You can't enqueue things if requests aren't getting through at all. Same deal with authn/authz issues, which they also had. If you can't answer the question "is this message allowed to be added to the queue" you can't enqueue stuff.
GitHub does use queueing for all kinds of stuff internally, though, because they're not morons.
The comments just shows how entitled people have become. Most people use GitHub and features for free and have the audacity to complain.
The outage is due to massive load increase. In 4 months the number of commits doubled to 2.9 Billions. Anyone worked with high load systems knows that’s it’s not a normal growth and how difficult even to keep on horizontally scaling in a short time period such a complex system.
GitHub should charge at least maybe 5$ monthly fee and most of the entitled freeloaders would leave the platform and it would free up resources
Using a corporation's free offerings isn't freeloading. Microsoft wants people to put their code on GitHub. They want GitHub to be the place where source code is hosted, it is incredibly valuable. Saying "GitHub is sucking and I might leave" is information that Microsoft wants to know if they want to preserve GitHub's dominance.
Of course some people take it too far. Of course there are reasons that the outages are occurring. But Microsoft wants GitHub to be a core, reliable pillar of the software world. Nobody's making them do that, they do it because it's good for them.
We're still staying on Github at work, but have had backup self hosted git repos as a break glass option when Github is completely broken and leveraged this several times now.
That sounds unusual for a free platform (not a limited trial but an actual free tier). Isn't it usually the case that only some small percentage can be convinced to pay?
I mean, a lot of us have paid GitHub a lot of money for CI on private repos. And when GitHub themselves encourages the insane behavior of vibe coders and agents instead of just charging or rate limiting access of bots, it's hard to give them sympathy.
I strongly disagree. GitHub, a year ago, acknowledged the fundamental problems and began work on them. We all agree with the diagnosis and strategy: stop building new things, bring stability. Why would whether the CTO codes have any bearing on the correctness of this strategy? GitHub’s problem isn’t that leadership don’t understand the product, or that they don’t know what they should be doing, it’s that they’re battling unprecedented demand. If it was a disconnect between users and leadership on what matters, sure, a CTO who doesn’t use the product would be notable, but that isn’t the problem. And that’s all assuming he doesn’t actually use the product, maybe his privacy settings hide private commits.
In other words, I think his private contributions should still manifest on the contribution graph. And for being the CTO of an organization like GitHub, with no open-source contributions... Not a great look.
Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.
> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.
If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go great. But there are lots of situations where one or both of those breaks down.
In GitHub's case this is especially relevant - the only reason to throw an error message at the user is the hope they - the human - give up and walk away (or you break all the CI/CD builds and the time it takes humans to hit "retry" gives you some breathing room).
I feel like I'm mildly stupid in trying to out retries as heresy but I'm not sure.
You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.
The dev system we use for a 3rd party hosting provider (a big one) requires a commit and push for every file save while we're developing. I created a build system for this that copies the whole repo to a temp folder. As we save changes to files in the main repo folder, the build system watches for changes and copies the changed file to the temp folder, then does a commit on the temp folder and pushes to a an intermediary repo in github which then triggers an action that causes the 3rd party system to update from the intermediary repo. This way we don't pollute our main source repo with a commit every time we save an update to a source file.
It's not my favorite way to develop but it's caused us no real problems except when github goes down.
Microsoft and GitHub's only option is to suck it up, absorb this growth, and lower failure rates. They have the money, so that's not the issue.
As someone on the sidelines, this is really interesting to watch unfold.
If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway.
If alternatives aren't able to manage, then user's aren't going to jump since those services won't actually provide more usage.
Imagine a fee over X commits, but only during certain hours. I can imagine 90% of the commits over 6 or 8 timezones, maybe 50% over 4 right now...
I had an interview with someone who has refused to use AI, and he has been unemployed for 2 years since graduating.
On the bright side, if my 6 year old can make 3D video games, I expect seasoned programmers to be able to pick it up quickly. I think ego is the hardest thing to break.
edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.
Useful / impressive for whom is the question. Not for us!
We pay for Github enterprise, and because GH can't be bothered to separate service tiers for sloplords and actual paying customers we get garbage level performance. They could of course always implement usage limits, but the goal is not to earn money, or provide a good service, the goal is to maximize AI users. Would be very awkward at the next executive golf meetup if you couldn't point to increased AI adoption.
In short: This is why monopoly laws matter. Once a company becomes too large, normal business rationales cease to be the motivation for their actions, and GH can go along with the pied piper of AI psychotic C-suite officers like MS is doing instead.
Try spending $20/mo on gpt sol and combine it with OpenClaw.
Seems false. Lots of coding adjacent people, engineering managers, etc. are now pushing PRs.
> How is it impressive if we all know it's autogenerated?
Nobody is saying the code is impressive, just the growth of github is impressive. It's not doing anythign different based on the source of the code.
Not all growth is good, especially growth that is actively hurting the company.
There is an equilibrium in both nature and software. Purposely designing systems that mimic the effects of cancer is going to benefit who exactly?
Github isn’t small startup, where other 10x threshold is as cheap as buy bigger box in your IaaS.
When you are already biggest player in the ecosystem and you suddenly get 10x persisted traffic, with at least 30x+ forecast “soon” - I am not surprised they have issues.
[0] even at current MS owned github
I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.
I assumed it does, do you know that it doesn't?
Running a GitHub clone at their same scale as a customer on cloud pricing would likely be insane. But y’all know infra is actually quite cheap when you run it yourself right?
It’s usually the case with these M&A deals that the profit just never quite makes sense to justify the purchase price, unless you can truly scale up the user base or revenue model. GitHub was already so mature as a solution when they bought it, I don’t know that they could have added that type of value just by slapping a Microsoft logo in the footer.
It feels like GitHub maybe needs to slow down? 'We must change things faster' is a wild way to start off an eight hour hard-down postmortem.
The worst outages I've been part of always have some version of this :(
I suppose the main reason I think it might be a bad idea, is that it would add complexity to the reverse proxy (i.e. now it's having to track whatever thing is being used and that complexity itself becomes a potential failure point...)
(To be clear, the clients should have their own backoff procedures, but I'm thinking about cases involving naughty clients, which are sometimes a harder problem to correct for...)
Do you know of a documented instance where it would have helped?
If you have, say, a long poll then kick off all users due to a deploy or error (or a broadcast message) then you can have a situation where you’ve got a huge clustering of connections at 1 minute, which spreads very slowly out as real life issues give you jitter for free. You can avoid this or at least return to normal much quicker by adding some jitter.
It might happen if all your users back off at the same rate too, if the clustering causes a bunch of errors. Error -> lots reconnect 1 minute after -> fail -> lots reconnect 2, 4…
More likely to occur in cases where there’s a way you can have people all connecting at the same time - synchronisation to a real world event is one case and then connecting again at the same time after.
I felt dirty implementing such a solution (introducing randomness for the sake of randomness is off-putting), but it worked. It wasn't until a while later that I even heard the term "jitter" in this context and realized this was a pretty decent solution for this kind of problem.
Regardless, if you're going to introduce something where adding jitter is appropriate, I'd just just add jitter. It's not premature optimization; it's an essential part of that kind of functionality.
Downstream database of our edge serverless platform went down. A tonne of requests failed all at once. Every service in the microservice request path, and the client, had their own retry policy.
Clients all retried at the same time. Retries amplified in our microservice graph (1 request at the front door ended up with like 10s of retries internally as each downstream microservice along the path retried requests). Request queues backed up and couldn’t drain fast enough. Clients all timed out at roughly the same time. All waited the same time. All retried again at the same time.
It was a pulsing thundering herd of many hundreds of thousands of requests at the front door that was amplified by internal retries.
Had to tune up load shedding to 100% after the database outage was mitigated until the backend recovered then tune it down in increments to restore service.
Added jitter to clients and turned off retries on the serverless platform.
https://buildkite.engineering/sleeping-at-scale/
It is specific and technical.
What I read from this is: * Scaling is hard, we don't have enough capacity * We give away a shitton of compute for free * I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.
Notice there's nothing about paid customers, I'll add in what they are missing:
Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.
> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations
How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.
Well that's because in comparison to the absolute flood of traffic brought on by AI, they really haven't been operating on this scale before.
Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem
GitHub outages seem to be very normal and even more emotional these days. Yet, that’s no reason to treat each other destructively. It’s sad.
So let’s turn this energy into creating something better.
I don't like paying for free stuff but gh certainly worth it.
Might force people to review their slop before pushing it.
But I'm not going to pay per commit over my monthly plan
I said in another thread that they can't blame increased demand for these outages, but the demand growth is genuinely insane for a company already operating at huge scale.
I guess we'll have to wait and see if they deliver now, but it seems like they're taking it seriously at least.
GitHub has alternatives, but no replacement
https://news.ycombinator.com/item?id=49135365
Why developers are ditching GitHub for Codeberg and self-hosting alternatives
https://news.ycombinator.com/item?id=48842611
and new entry:
Cursor Origin Code Hosting
https://news.ycombinator.com/item?id=49334209
Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.
Github is ripe for disruption and I hope it is disrupted soon.
And from the RCA [1]:
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.
[1]: https://www.githubstatus.com/incidents/zkxwbgr0cnmx
It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage, unless they have no reporting or their tooling is completely incompetent. If things can fall apart so easily, throwing more capacity at the problem won't fix it.
If it was failing slowly, why didn't they fix it then? Why didn't they notice?
Could it be because they fired all the people who know how to monitor and identify such problems? That's what SREs are for.
Please share the information you have which contradicts the conclusions I have drawn from Github's statement.
(And we know they're liars. They report very few of the actual incidents they have; see for example https://mrshu.github.io/github-statuses/)
> It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage
In my experience, there are hard thresholds that get passed that expose hidden bottlenecks like this. A previous system I worked on we had absolutely loads of headroom by all of our measured metrics, but one day we filled a cache because the value hadn’t been tweaked in recent memory. Plenty of space on disk and in memory, but all of a sudden we went from a very high cache hit rate to a very low cache hit rate, and everything ground to a halt.
These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.
The art of large system design is to identify and avoid these kinds of chokepoints. And when something happens, propagate the "backpressure" up the stack to avoid queuing.
AWS got a fair share of similar outages, so the newer SDKs now try to not exacerbate these kinds of issues: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retr...
The original AWS EBS outage is probably the canonical example: https://aws.amazon.com/message/65648/
Look, I hate Microslop as much as anyone but you'd have to purposely misinterpret TFA in order to arrive at this interpretation. C'mon.
for context, I would GLADLY move there my Neovim plugin. all it does is brings the current jj message into your editor and lets you integrate it with a status bar (or anything in nvim, really). that would be a decent measure against drive-by slop contributions, and I'd accept contribs over private github mirror from those who I know but can't bother setting up git mail
EDIT: TIL that one can host SourceHut themselves. discoverability may still be a problem (sr.ht just ranks higher in search engines) but 1) fixable with github mirror that points to sourcehut instance as a canonical development platform, 2) it's moderately easy to sync contributions between tangled and sourcehut, so tangled is also an option
EDIT 2: the email part would be PITA, so $4/mo is attractive on that background
If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.
My personal GitLab on the other hand really does take only a day or two per year.
That said, a week or two per year is just what it costs to maintain any one thing period. I spent about that much time maintaining PCs in the office, or my personal proxmox setup. It's not onerous at all.
GitLab is super bloated and a little sucky to admin, but it's not too bad all things considered. I'm admin in my new job's GitHub org and it sucks a whole lot more to maintain.
Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but it's unattractive to me - as someone who agrees with most of their current politics - because the day they decide they don't like me, I'm screwed.
I've actually worked with a couple of companies who do use GitHub for their code, and they all use Linear in addition to GitHub.
I understand the concern you're talking about wrt. Codeberg, but I wouldn't view it as a significantly bigger risk than anything else. Any platform can suddenly decide that your project is against ToS (GitHub will absolutely not accept just anyone's use of their platform either) and Codeberg introducing some rules recently doesn't, in my mind, drastically increase the risk of a dramatic ToS change in the future. But we all have to make our own risk evaluations and I won't judge yours. Luckily, moving between Git hosts isn't that difficult; setting up CI again and losing merge request history does suck but it's not the end of the world, unlike something like, say, losing your AWS/GCP/whatever account.
Lots of companies moved because it was cheap, but it's not anymore. Ironic that companies might choose to migrate to them now for stability, rather than price.
Also for accuracy, Codeberg has a pro-human and anti-corporation stance. Both of which are definitely en vogue at the moment.
Much better than GitHub's pro slop sentiment, which is doing nothing but destroy their reputation.
It's an expensive, low revenue generating site.
There are, and have always been, competitors, including "host it all yourself" solutions, but nothing has really stuck.
How is it "ripe" for disruption?
That can't be cheap.
Compute and Storage for Free Tiers: Hosting code for over 150 million developers and processing over 2 billion GitHub Actions (CI/CD) workflows a month requires astronomical server power and data storage. The "Free" tier is a massive cost sink that Microsoft treats as a loss-leader marketing expense
Let me know when you understand how that's not free.
GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.
I self-host it and its pretty easy to have uptime higher than github when you have 3 users total.
Absolutely do not recommend using GitLab.
That says absolutely nothing. The "What is Forgejo?" question is unanswered and instead you get a lot of words about their values, their inclusivity, etc. And the next thing in the docs is how to install it. It's ridiculous.
I still don't know what it is or what it does.
You're in luck, GP comment described it for you.
self-hosted > You install it on your server
lightweight > It does not consume a lot of resources (cpu, disk, ram)
software forge > offers tools that help with creating software collaboratively (repository hosting, change request management, wiki for docs,…)
https://github.com/google/git-appraise (unmaintained)
You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.
6 years ahead [1] on not going all in an centralizing everything on GitHub.
[0] https://www.githubstatus.com/incidents/bhbcjn4n3jzp
[1] https://news.ycombinator.com/item?id=22867803
"If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.
I’m done.
Seems like we are headed for Tech Gridlock.
I don't think GH has any of those, but just keep scaling, but that scaling failed.
Just bad architectural decisions from the postmortem.
--
It will only get worse due to AIs spawning massive commits, and they don't have unlimited cloud resource.
They can scale but not scalable in terms of effort, resources, and $
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.
GitHub does use queueing for all kinds of stuff internally, though, because they're not morons.
"You've seen what we've done. The August 21st outage comes next. See you then!"
The outage is due to massive load increase. In 4 months the number of commits doubled to 2.9 Billions. Anyone worked with high load systems knows that’s it’s not a normal growth and how difficult even to keep on horizontally scaling in a short time period such a complex system.
GitHub should charge at least maybe 5$ monthly fee and most of the entitled freeloaders would leave the platform and it would free up resources
Now remind me again, who trained a coding-assistant without consent on those "freeloaders" code and sold it for profit?
Of course some people take it too far. Of course there are reasons that the outages are occurring. But Microsoft wants GitHub to be a core, reliable pillar of the software world. Nobody's making them do that, they do it because it's good for them.
We're still staying on Github at work, but have had backup self hosted git repos as a break glass option when Github is completely broken and leveraged this several times now.
It sounds a little bit unfair to me.
Do you have a source for that factoid? (I suspect the vast majority of Github resource usage is paid. And we are upset.)
It's a service that is owned and operated by Microsoft Corporation, and we're the product of it.
People take the weirdest rhetorical hostages.
https://imgur.com/a/zIbT0Gi
It shows zero contributions in the past year, on this account. This is a huge, huge red flag.
Here's me: https://github.com/ryanisnan
In other words, I think his private contributions should still manifest on the contribution graph. And for being the CTO of an organization like GitHub, with no open-source contributions... Not a great look.