(The latter has a fantastic worldbuilding/intrigue system, but it's combat system is basically pew pew back and forth til someone's defeated, and it's just not cinematic or dynamic enough for the folks I roll with. I kinda want to bolt a better combat system onto its social system.)
You can definitely do this. I might wind up doing it with my upcoming sci-fi campaign. I just bought Traveller and Stars Without Number. I haven't read either book yet, but I'm planning to run a combo of both. Likely SWN's combat and leveling system and Traveller's everything else.
I can run a really bonkers 5e campaign with minimal prep now because I know the system like the back of my hand, and I improvise story beats in a way that has the players thinking I planned it all out months ago
A few YouTube DMs I watch recommended this to strengthen your improv muscle. I'm confident I could do it, but I'd be sweating bullets the whole time.
I think if a table of folks all agreed to learn the rules together I could do it, but, again, love my players, they never opened the player's handbook, and they aren't going to start now, and that goes for new games, too. It's all on me to onboard them.
My players dislike my penchant for new game systems, but fortunately, most of them dive head-first into the book when I force it on them. I have two mini-DMs sitting next to me by the time I start the game.
I like a lot of modern games (by a certain definition of modern), but I'm old enough to feel nostalgic for when D&D was set up for different jobs requiring different classes. Back when success was measured by recovering treasure, you could have a class that just fought or one that just picked locks and so on. I think once we started measuring classes solely by how well they killed, we lost a lot of that distinction.
I'm a known D&D 5e hater. I like the built-in flavor that comes with modern D&D classes and subclasses, but you and Damien are right that the classes have become samey. You need a rogue to pick locks for you. Unless you have a bard. Or a ranger. Or any other class with proficiency. When everyone's a specialist, no one's a specialist.
The long-term campaign my group wrapped up last month had five players, and four of them could've been the "face of the party." We had a warlock, bard, paladin, and charisma-based rogue (me). When literally 80% of the party can serve as the charismatic frontman, something might be wrong with your game system.
There's something to be said of OSR. If you need a sneaky guy, you need a rogue. The rogue couldn't double as the party tank—something I often did in that long-term campaign, by the way. I had more HP than our fighter.