Revolut says that in less than a year it’s managed to slash its [code generation] costs by roughly 90 percent by making self-hosted models its default instead of relying primarily on commercial services like Claude.
But they are using Sam’s RAM while doing it 😡
Spoilers: it’s Actually Indians.
Oh that’s why an hacker managed to phish them 680 times to get all details on his victims.
Passports. Documents, transactions, and so on: the hacker pretended to be an Italian police officer and asked to get info on millionaires, an LLM believed that without question 680 separate times.
Not saying it was ok, but the issue here was that the Italian police got hacked not revolut, from their pov the request was valid.
Still, they should have evaluated if it was legit and not just rubber-stamping hundreds of requests in a short time from a small police office in Southern Italy that would send 1/year at most. And think “why this Italian officer wants info on finland citizens?”
That I can agree to, I suspect plenty of it is automated or the people will be under so much work that there are no incentives to do so.
Self-hosting is illegal in 5 … 4 … 3 …
You wouldn’t download a plagarism machine!
If AI isnʻt going to go away, it needs to be cheap, sustainable, open source, and self hosted/ hostable.
And ethically trained. People should be financially compensated if their data was used to train AI without their consent.
I don’t think it’s really possible to do this.
I mean speaking just logistically, how would you determine who gets how much? If it’s based on quality, how do you determine what is of high quality? Otherwise, does everyone who has every written anything that finds its way into a training dataset get like a tenth of a cent?
Alternative proposal: all AI models must be released as open weights, available to everyone.
AI, through its use of training data, has benefited from the sum total of human knowledge, therefore the results of this training should be given back for the use of all humanity.
That seems fair to me.
And they could keep exclusivity for some time then have to release the weights
Patent-like style, so a company gets incentive to make better models
This is also my opinion. I always advocate for putting GenAI models under UNESCO stewardship, because they are a snapshot of all digital, cultural output the world has to offer, and therefor they aren’t less important than physical cultural artifacts. Free personal use, licensed use for commercial usage, with lower prices for educational/nongovernmental organisations. Use the funding to provide grants to the arts in a large scale.
I think a sensible place for this would be the base models, those are the ones that are trained on the entire internet in order to get a good working model that “understands” things. Then leave the fine tunes on those base model commons as available for companies and organisations to bring their own value to.
Holy shit! What an excellent idea.
I always advocate for putting AI models under UNESCO stewardship
As now will I.
Not only open weight, but the dataset used should also be public.
You are right if we look at it short term, long term llm will be replaced with the next thing and who knows maybe it can be trained ethically, at least we should require research into it.
I don’t think it’s really possible to do this.
Why not? Both parties have to figure out how much they want to pay or to get payed. Similar like it works for any other license like on Netflix or Spotify. It is not the AI companies job to determine how much they will get payed but how much the data is worth to them and the data creator how much they want.
If i am at a bakery and the bread they offer does not look like its for that much to me then i have not to buy it.
How do you suggest an individual should go about this?
Do I go down to my local courthouse and file a suit against Anthropic so I can ask the judge to subpoena them to determine if my Reddit shitpost from 2012 is in their training set?
Class action lawsuits. How would something like this work for anything else? Like Netflix subscription violations?
And we are talking about meaningful data not your shitpost, like licensed code or a book you wrote and anthropic pirated.
And we are talking about meaningful data not your shitpost, like licensed code or a book you wrote and anthropic pirated.
Ah ha! You see, that’s the problem. My shitpost is meaningful data. This is something people seem to have a really hard time understanding. Scaling laws and the size of the search space in AI training means that the datum is not valuable. It’s only the data (in conglomerate) that is value for training, and my shitpost would be a part of that.
Once you understand this, you should see what I meant when I said it’s not feasible. How do you determine and then fairly distribute the payout to everyone who has ever written or recorded anything in digital format? This is not a simple supply vs demand equation, like your bakery example assumes.
Also I have no idea what Netflix has to do with this discussion.
Ah ha! You see, that’s the problem.
No. Meaningful in the sense of copyright. Data you get copyright
There you got your distinction.
Private copying levy was implemented to compensate greedy media industry. This was included in the price of every writable massmedia device in Finland until 2015.
Maybe something similar could be used with AI. Directly compensating for the victims of ip theft would be pretty impossible but the money could be used to fund art programs and such. AI companies would be the ones paying, based on the usage their models have, be it online or self hosted.
At least in my country, business can ask for a full refund of that unjust tax by declaring the drives are not used to store pirated movies/songs
what about open models, and the proportion they should pay in each country…
Eh, not sure if it will work
That soup is already cooked. And obtaining consent is prohibitive, so your data set will be tiny.
Open weight models cannot do this, they do not have any funding. If you demand this, you are just demanding that there will never be any open models, that proprietary AI models controlled by massive trillion dollar corporations will be the only models available, and frankly the cure is worse than the disease.
It’s also impractical, like closing the gate after the horses have already gone. Open-weight, unethically trained models are already out there. They are not going away. You cannot put the genie back in the bottle. They will never earn any money, so they will never be able to compensate for the training that was done to create them, unless you consider the fact that they exist, and they are open to anybody, as at least some measure of compensation.
It’s not enough compensation, but it is something. The training was certainly unethical, but so is an economic system that requires intellectual property to be transactional. Why fix one unethical thing while keeping and reinforcing another?
The only sensible and reasonable way to resolve the issue in the long term is for significant government support and patronage of the arts and content creators (and ideally all people) to be established. People who do this for a living are providing valuable service to all of humanity and that deserves to be recognized not by transactional, metered capitalism that defines each piece of art or content in terms of its raw economic value (a system easily and frequently corrupted anyway if you look at the prevalence of theft, forgery, fraud and money-laundering in the high art world), but by guaranteeing a comfortable and dignified life that allows creative people to freely create the art or other works they love and feel passionate about without worrying that they’re going to starve to death or be forced into a dead-end job if they’re not economically successful.
There’s actually a reasonable argument for global regulation of AI.
Every country faces an economic doom loop as AI becomes more capable, once it is capable of automating away a sufficient amount of labor the economy will spiral. This means every country, in a game theory sense, is facing a no-win prisoners dillema. That is precisely the recipe that allowed for nuclear nonproliferation.
It is completely within the realm of possibility for the international community to agree to control the training of models at the national level and heavily regulate distribution commercially, similar to what we do with nuclear power.
It is, but the issue is the right time to buy hardware was 2 years ago.
Its like 1/ Mooreʻs law.
The right time to buy hardware was always 2 years ago.
This is going to be a trend, and why I think the IPO for Anthropic is ridiculous. Competition is massive, there is no moat.
Any sensible company, that has the captial, is hedging their bets. My place has plenty of Claude and Copilot licenses, but we also have healthy local projects going on to.
Personally I can use Claude if I want, but I tend to use the self-hosted platform (Qwen and GLM on DGX Sparks before anyone asks). Big Boss told everyone to be sensible with expenditure, so this part of me doing my bit.
Before anyone comments about “is Big Boss also saving money?”. Yes he is - the guy comes from a French farming family, he is flies economy out of choice and is well-known in our industry for sacrificing not just his bonus, but also his salary when the company has been in rocky waters over the years.
It still makes sense for very small teams where the burden of maintaining the infra is too high, but if you have 2k engineers it’s a no brainer to self host. It’s only a matter of time before open weight models can process vision as well.
I think the doomerism was all to get that moat. There’s no vendor lockin either, even if you’re using the API. You can switch between providers really easily.
You can run litellm as a proxy to not even worry about changes in the API (much).
Not to mention there isn’t public pushback against an office with a server room like there is against dedicated data centers. Assuming they are running servers and not just some desktops with GPUs.
About time this hit enterprise.
Been saying the downfall of OpenAI and Anthropic will be local models that can accomplish the same thing without needing a dstacenter worth of GPUs to run.
Plus you have no total token limit on self hosted models. You can let a slop agent run for weeks without ballooning costs or getting rate limited.
And no model censoring, so you can go build that nuclear bomb you always wanted lol.
Both AI giants are already losing a ton of money because their pricing isn’t viable for the amount of training they’re doing trying to be on top. If they readjust, they’re gonna suffocate against much cheaper solutions.
You totally have a limit. It’s what the cards can output.
Revolut has 54x B200 cards (according to their video on this). So their limit is whatever that can output each month
I’m not surprised it’s moving this direction, but I am surprised it’s moving this direction now.
Hardware is extremely difficult to find right now and token prices are falling off a cliff in terms of price per intelligence. About 6 months ago I was talking about how it’s only going to get cheaper from here and still seems to hold up today, we really don’t need anything better than Opus 5.5 and compute is only going to keep increasing (price decreasing) who knows what the field will look like next year.
Either way I’m happy to live in a world pushing towards that, just surprised.
The data center is still needed, just not the same scale. You don’t run “small” local models to compete with Claude’s size. They don’t publish numbers, but estimates were like 1-2T params for the models like early Opus. You need a lot of hardware to run those efficiently at a scale an enterprise would need.
Or they just spent enough to upgrade developer desktops with RTX Pro’s to run quantized ~100B param models at their desks. Those are very good for coding.
According to them they host 54 B200 cards to do code generation. While claude etc does the high level planning and design
Microsoft predicted this some time ago which is why they have begun pushing a Hybrid Model where tasks are run locally first and only going to the cloud for heavy lift.
You know what… I like this because it pisses off Sam Altman
tbf every business does use AI these days anyway, it’s vibe coding that’s the issue.
The issue with AI is not the tech itself, but the people controlling it, so I definitely agree with you!
That it’s in the hand of a few people is one major issue, but the externalized costs* are even worse one in my book and this is another post where it’s conveniently ignored mostly.
- Energy consumption, water usage, scraping ddos
It is quite concerning, but I feel like those issues have a solution, if the tech moves out of those greedy bastards hands.
They just want to scale as fast as possible without any concerns for communities, energy, or water. That’s because they are running a completely unsustainable business and they need to spread quickly, to make more people depend on the tech, as provided by them, so that eventually they get government backing and become comparable to energy, as they have said before.
But it’s a fucking scam. We could run AI locally with less resources per area, so by distributing it the impact would be negligible compared to what massive data centers bring. I would actually love to see some numbers on this, but the ones funding it are the ones interested in building the data centers, and they have the government reach to keep them from doing a fair study as well.
Yeah, also how it’s used. If revolut knows what’s happening in their code, then should be grand
Oof, remind me never to get an account with Revolut.
so they are paying -9x now?
Yep, but obviously with a big initial investment.
deleted by creator
