We just reported about the Linux 5.12 changelog with a focus on Arm, MIPS and RISC-V targets on Tuesday, and at the time, the expectation was a delay of about one week after Linux 5.12-rc8 was outed on Sunday, April 18.
But Linux 5.12 could be further delayed due to shenanigans from two Ph.D. students doing a research project on open-source vulnerability at the University of Minnesota. This was announced by Greg Kroah-Hartman on the Linux kernel mailing list.
Commits from @umn.edu addresses have been found to be submitted in “bad faith” to try to test the kernel community’s ability to review “known malicious” changes. The result of these submissions can be found in a paper published at the 42nd IEEE Symposium on Security and Privacy entitled, “Open Source Insecurity: Stealthily Introducing Vulnerabilities via Hypocrite Commits”
So their work at to be reverted with 190 reversions so far. It also means future commits from the University of Minnesota will be rejected from mainline Linux by default.
You’ll find the research paper hosted on Github. The research paper mentions targets for “hypocrite commits” include large projects that accept commits from third parties including the Linux kernel, Android, FreeBSD, Firefox, Chromium, OpenSSL, and others. But it appears their Proof-of-Concept experiment only targets the Linux kernel using UAF (use-after-free) vulnerabilities as they identified over “6K immature UAF vulnerabilities in the Linux kernel that can potentially be turned into real vulnerabilities”.
The paper also implies they did so ethically and informed the maintainers as soon as one of their patchsets was confirmed.
Our goal is not to introduce vulnerabilities to harm OSS. Therefore, we safely conduct the experiment to make sure that the introduced UAF
will not be merged into the actual Linux code…
Once a maintainer confirmed our patches, e.g., an email “looks good”, we immediately notify the maintainers of the introduced UAF and request them to not go ahead to apply the patch.
…
All the UAF-introducing patches stayed only in the email exchanges, without even becoming a Git commit in Linux branches. Therefore, we ensured that none of our introduced UAF bugs was ever merged into any branch of the Linux kernel, and none of the Linux users would be affected
They also wrote they point out the bug to the maintainers and provide the correct patches. But after reading Greg KH email I don’t think he shares the same feeling. It could be due to miscommunication, we just don’t know at this point.
Evaluating the security of open-source projects is a commendable initiative, but testing is certainly an issue, as it can’t be done by informing maintainers that you’re about to introduce “hypocrite commits” in advance. I don’t know the right answer, but if you feel you do, please feel free to comment below.
Thanks to dgp for the tip.
Jean-Luc started CNX Software in 2010 as a part-time endeavor, before quitting his job as a software engineering manager, and starting to write daily news, and reviews full time later in 2011.
Support CNX Software! Donate via cryptocurrencies, become a Patron on Patreon, or purchase goods on Amazon or Aliexpress
I think it is ok to ban University of Minnesota for 10 years as a standard procedure when trying to insert bugs/backdoors into a project. Malicious developers are just as bad as the incompetent ones. Why keep them around?
They were not malicious.
They were testing defenses, which is important in order to catch actually malicious people.
The way they went about it was totally wrong.
All of the thousands and thousands of reviewed patches already contain the data they needed. There are going to be plenty of commits that add memory errors (I added one myself during 5.12) and a lot in the exact class they were targeting of commits that claim to fix memory errors but actually created new ones.
They could have made their hypothesis and then written the analysis tool they claimed to have to go back through the git history and kernel lists to find commits and patches that fit their hypothesis to try to prove it.
They would have been able to prove their point, not piss off everyone by wasting their time, not have potentially added bugs that could result in real world consequences etc etc etc.
Totally agreed, it’s clear they were seeking immediate fame through a lot of buzz. It’s a success, I didn’t know that this university existed, now it’s known all over the world!
the correct way would haven to approach someone like GKH, LT, … and figure out a procedure. it’s not that GKH or LT are the maintainers for the entire kernel. but if GKH, LT, … aren’t open for such research shame on them.
You can’t do this research by telling Linus and/or Greg that you’re going to send dodgy patches on purpose and to not accept them from sub-system maintainers if they end up in pull requests.
Do you think anyone would trust them again if they let a bunch of researchers use them as guinea pigs? Imagine you spent your free time reviewing patches, thought everything looked in good shape and then when you sent your PR to Linus he said “hah! you let through one of the dodgy patches!”. For some people that would be decades of trust down the toilet.
American hackers trying to destroy the largest international community piece of code :(. Could they be related to some of their government stance, that try to add security breach everywhere as possible?
Although it comes from an American university, the nationality(ies) of the students is not clear.
Name them: Qiushi Wu and Kangjie Lu. Kangjie Lu is an assistant professor.
I had this suspicion, and it turns out their nationalities are exactly as suspected!
The person who sent the “slander” message to Greg KH was Aditya Pakki.
“I didn’t know that this university existed, now it’s known all over the world!”
They kneeled on the neck of the Linux kernel.
Yesterday I read alot of the thread on lkml and to be honest either those guys are extremely incompetent or extremely dishonest or any combination of both…
You convinced me. Unless they found some new method disgusting the malicious code, test with the same thing has been repeatedly detected, is wasting everyone’s resource.
Honestly I think they where partnered with some government entity. Intentionally adding any bug is wrong and ethics is the most important thing about security and IT. Please just ban that college altogether for the future of Linux
They fully knew what they did and did it again. I mean come on people like this is why Linux is so broken.
I don’t believe there is anything ethical about the actions of either side. An email stating the premise of the attack would have sufficed. The demonstration of it has eroded confidence and cast doubt on ALL past commits from anyone. The unilateral “banning” of an entire organization is collective punishment[1] and affects the innocent far more than the guilty.
I doubt the actors were the first, nor will they be the last, to implement this type of attack. I would rather hear how Greg intends to prevent this from happening from non-University of Michigan contributors. I fear the real answer is “he can’t”.
[1] https://en.wikipedia.org/wiki/Collective_punishment
I don’t think this will cast all past commits into doubt. I think it will make people suspect one off patches, huge batches of machine generated patches, patches from totally unknown people, patches with bad/vague commit messages etc.
If it stops people like that guy that sent out hundreds of small typo patches that were not formatted correctly, added new mistakes etc etc then I think that’s actually a good thing.
I’m pretty sure Greg’s ban isn’t a permanent ban and more like “until we can be sure otherwise we should not accept anything from these guys as they can’t be trusted”. They need to rebuild trust and IMHO they will need to do something better than the sort of machine generated fixes they’ve been spamming out to do that. Also I’m not sure what gives Greg the authority to ban anyone. Surely it’s still up to Linus if he actually wants to merge a PR.
And anyway from day one the principle of contribution has always been based on trust and reputation: if you appear trustable we can work with you, if you betray us, you ruin your reputation. This is as simple as this. There’s no reason for this to change as it remains quite solid. OK these guys managed to introduce crappy commits. A few weeks later they’re reverted, end of the story. Linux will continue to progress. However when these guys leave the university, they will have some fun trying to find a job. They’ll probably start a “security” startup like all those who believe they can fix the world, and 3 years later will end up selling cars or appartments.
> However when these guys leave the university,
> they will have some fun trying to find a job.
I personally don’t wish any long lasting consequences on them.
Everyone should have a chance at redemption and all that..
It would have been nice if they had replied to Greg properly, admitted they had messed up and then helped to clean up the mess they caused.
Right now it seems like the community has to wade through the series of 100+ reverts Greg sent so even more time is getting wasted.
I guess their university has banned them from touching a keyboard at this point so maybe they can’t help even if they wanted to.
> I personally don’t wish any long lasting consequences on them.
> Everyone should have a chance at redemption and all that..
Sure, I agree, at the same age many of us have been doing stupidities like this. I’m not proud of my old findings on bugtraq 25 years ago pissing off people with zero day exploits by reporting issues without helping to get them fixed :-/
> It would have been nice if they had replied to Greg properly, admitted
> they had messed up and then helped to clean up the mess they caused.
Yes that’s the real problem, the overall attitude. If they had immediately told the few maintainers who merged their code “hey sorry, it was an experiment, please revert it now” then apologized on the list for having annoyed everyone with stupid code reviews, then shared an exhaustive list of bogus commits or useless ones, it would have gone way better. But there used to be a precedent with some of them repeatedly posting absolutely stupid patches that fixed “issues” detected by their home-grown code scanner which was apparently as reliable as a random generator, and the persons not willing to understand (which probably was the reason for Al immediately jumping on the gun). So I guess that explains why Greg plans to revert all that pile of crap as far as 2018.
I suspect the chief of the lab they’re working in wants to have his name associated with the awesome security discovery of the year and is using his students as cannon food for this until it eventually works. That’s a sad affair, really.
The CIA / NSA pays contractors to do this kind of dirty work since long, in hardware, OS and popular software. All spectrum dominance is their 1984 motto, also for the US itself. Leader of the free world my *ss.
Sorry, I can’t blame Greg here.
https://news.itsfoss.com/hypocrite-commits/
“This is not ok, it is wasting our time, and we will have to report this, AGAIN, to your university…”
The commits weren’t as stealthy as they thought. The University of Minnesota researchers had a chance to deescalate but started talking about “slander”.
U of MN students can probably get around it by using a different email address, or get a little history lesson if they find out they’re blocked. U of MN can negotiate if they want the ban lifted. Nothing wrong with this punishment.
Well this university seems to have quite a sloppy ethical commission and no guidelines when it comes to public communications, so why not banning them until there is an official statement from the university (no not from some dodgy students/professors)
“I fear the real answer is “he can’t”.”
Always fix the problems at the root.
Banning the university won’t solve anything.
Banning the university actually will be very effective to stop similar incidents, not just from University of Minnesota but all other institutions doing the similar thing.
The owners hated people pointing out there was not enough life boats on the Titanic.
I’m pretty sure the fact that there aren’t enough reviewers and every reviewers load is far too high has been brought up over and over again. It’s why there are efforts to get new people involved by hand holding them through submitting their first “fix typo” patch and there are so many bots trying to break the kernel to find human errors.
These guys highlighted the issue maybe but it’s not like there was secret plot to hid the fact that to maintain quality at the current pace it’s not just about big vendors throwing code at LKML anymore more. They actually need to do tat for tat work and review code their competitors are also throwing at LKML.
These guys apparently wrote tools to find the sorts of errors that could be exploited by bad actors.
They could have written a bot that watches the LKML and sent out mails for fishy looking patches, proven their point and helped. Instead they went the instant gratification route and made themselves look like assholes.
Remember professionals designed and built the Titanic and said it was unsinkable..
All the reactions now are doing is putting people off testing and reporting methods of attack.
If you pull the blanket over your head the bogeyman cannot see you, to get you. Theory of safety and security.
>All the reactions now are doing is putting people off
>testing and reporting methods of attack.
Oh shut up. No one has said the research is bad. Everyone has said the way they went about it was stupid and unethical.
They could have proved their hypothesis without interacting with the LKML. There is decades worth of data out there for them to work from.
They could have analyzed data from the LKML and then told everyone on the LKML that they found it’s possible people have snuck in bad code in the past and they would have be praised for their work.
>If you pull the blanket over your head the bogeyman cannot see you,
>to get you. Theory of safety and security.
Ok. So you are ok with me coming to your house and breaking off screwdrivers in your door locks to test them for security issues? As it’s security research you won’t mind me doing that right? You promise not to call the police right?
You wouldn’t prefer me to buy a similar lock, take that apart and then warn you that your lock might be vulnerable?
Please link us to something you’re working on so we can screw with it and waste your time in the name of security research.
pretty nice explanation of what happened:
https://www.zdnet.com/article/greg-kroah-hartman-bans-university-of-minnesota-from-linux-development-for-deliberately-buggy-patches/
The university issued a public statement:
https://twitter.com/UMNComputerSci/status/1384948683821694976/photo/1
It looks like they did this for months.
https://twitter.com/kengiter/status/1330649966214340608
That’s probably why Greg KH said enough is enough…
His rationale is analogous to a medical professional adding poison to a cure just to prove it can be done and then administering an antidote right before the patient dies. Then, he expects everyone to thank him for showing a vulnerability exposed by the trusting patient.
#Unethical
Well he doesn’t hand out the list of his patients, neither does he give the antidote himself, he just mentions the drug store around the corner will know what to do.
For months and it seems also past the communicated deadline of the “research” that’s why certain people on the lkml seem to suspect the clean up is part of another “research” campaign going on… Very fishy, very weird how they answer or rather not answer to direct questions on the list…
Ignoring the controversy, I found the paper to be very interesting and a worthwhile read.
Just delete it.
The person who basically accused Greg KH of slander is Aditya Pakki, different from the paper’s 2 authors, Qiushi Wu and Kangjie Lu. He appears to be a 2nd year PhD candidate.
Wonder if a grad student was “talking out of turn”…
If AI would improve quality of reviewed code (commits), then we (as global society) are prepared to delegate setting up human readable code sources to AI as well (only defining [syntax,semantics,array] of results anymore)?
If this would be a discussion between compilers and machine code, why an erroneous input occurred would be cleared in micro seconds and effective work on improved code structures would get more attention?
This was a study about human interaction?
AI doesn’t exist, yet. It’s just a marketing term for ML and pattern matching
If you think otherwise, please point me in the right direction
If you combine it with Blockchain that counts…
if its about implementation of logic’s structures, we probably could do a lot more of automatically organized source code, but e.g. NDA’s or other items from political, law related or social ranking dependent influences are less likely being easily to obey on statistical AI’s training or ‘in some way of empirical’ AI approach?
There are meanwhile www search engines rules that are weighed on their semantic probability, but AFAIK not transparently to users knowledge or even to monitor on influence to search results.
If AI’s would suggest variants for human reviewers to choose from, that could be approach towards (kind of multiple choice, that is, btw, not recommendable considering educational progresses) faster review periods, if aims for software (hardware) milestones were decided priorly?
( e.g. considering increase of usage of cookies dependencies on web sites within recent 1-2 year(s) and variety of layouts, order of decision, languages or code syntaxes etc.? )
having a button on search engine pages, for including or excluding usage of (semantic) AI algorithms for interest related results of their users, this would be an empirical, user concerned idea of weighing that interest on company’s side and showing difference in results on users/researchers side without compromising companies internal knowledge?
I think there are two different things here and people focus on the wrong one. OK, it was a mistake to do that in that way, that’s the one thing.
The most important thing though is that now it’s clear that anyone can push malicious code and create backdoors in the kernel. It doesn’t even have to be a single commit, those backdoors can be a series of malicious patches that can be pushed with many months difference in between, but combined are creating a backdoor.
So, it’s more important to keep the later rather the fact that those “researchers” behave like kids.
Nobody really focus on how to make the kernel secure and create tools and processes that can find malicious code and backdoors that is already in the kernel.