From a game design “formalist” point of view, they are not different. A rules-centric view of games doesn’t care whether the interface is computerized, mediated via apparatus, or physical, so it makes no distinction between computer chess and physical chess; similarly, it makes no distinction between the rules of, say, baseball, implemented within a computer or by players on a field. They’re both still recognizably baseball. You can diagram them; you can port the higher level rules between media; you can implement even a phsyical version with a ruleset that requires everyone to play on their knees, or in wheelchairs.
The major distinction arises with subgames and interfaces present within the rules. For example, baseball-the-sport makes use of extensive implementations of physics, thanks to the real world providing a very robust physics engine. It also has a very rich set of subgames regarding mastering the controls of the human physical body. Computerized baseball is relatively limited on that front, mostly requiring mastery of just your hands as they manipulate the controller.
Sports historically refers to physical games, but of course even many non-sport games have large physical components involving either strength or dexterity. Many children’s games, such as jacks and tiddlywinks immediately come to mind (not that jacks was always a children’s game…).
Similarly, many apparently physical sports are heavily mediated by technological objects. At the highest end perhaps are various forms of car racing, but it’s a sliding scale downwards from there through bicycling, golfing, tennis, the effect of swimsuits or sneakers on physical capabilities, and so on down to a true “naked” sport (of which there are actually very few).
Etymologically speaking, “sport” being associated with physical activity is a 500-year old usage tacked onto a 1000-year old word root. Like other such words (“fun” being an obvious one) it therefore has very fuzzy boundaries.
So, from a game designer’s point of view, they aren’t that different.
Some theorists use the competitive nature as a way to classify games that fit the description; under this definition, backgammon, Blokus, Monopoly, and Pong are sports. But better words exist for this sort of thing, such as “contest” up through “game” (used by Keith Burgun) or “orthogame” (used by Elias, Garfield, & Gutschera [affiliate link]). This is additionally complicated by the fact that it is remarkably easy to turn a non-competitive game into one simply by measuring players asynchronously (high score tables) or in parallel (as is done with footraces). It seems to have become clear that the broad swath of games that can serve as contests is quite large, and therefore, the commonest division is actually “toy” (lacking goals), “puzzle” (has one predetermined solution), and “game” or “contest” or whatever, which is “everything else.” (And by implication, admits of strategy, typically unitary goals, multiple solutions, etc).
So… your answer is not going to be found in game design theory. Instead, we have to leave formalism behind and look at people.
In cultural practice, sports has come to mean competitive games played in front of spectators. It’s hard to think of a sport that doesn’t involve spectation. Even sport fishing is about relative competitive measures of skill and prowess in an arbitrary ruleset, quite distinct from fishing for food. The practice of snapping pictures of the size of the catch reveals that spectation is the driving force there.
Conversely, it’s fairly easy to think of competitive games not played as a sport.
Once spectation is involved, so is money. And so a sport is most typically characterized by infrastructure: tiers of amateur to semipro to professional; training; regulating bodies; teams that fans can ascribe loyalty to, etc. Culturally, some contend that sports are effectively safety valves sublimating the impulse towards warfare, permitting tribal affiliations to express themselves in a safe way.
The acceptance of eSports is largely due to the fact that competitive play of first-person shooters and RTS games acquired fans, an audience, and then venues and broadcast television channels in South Korea. In the US, a similar path was followed by the fighting game community, or FGC, albeit with different access to mass media, followed by FPSes and now MOBAs. The audiences for League of Legends matches now number in the millions, quite comparable to physical sports, and athlete visas have been granted for competitors.
On the flip side, there are many physical games that are not treated as sports typically; ringing the bell at a carnival is one example, despite the fact that it is not that dissimilar to a weightlifting competition (consider the difference between this, and say, the caber toss or other folk sports). With the increasing nichification of mass media, there is room for more and more sorts of sport activities to arise, and we have seen the development of numerous new ones in the last few decades. Some have been driven by new technological apparatus (better roller skates enabled new sports; same with skateboards). Some have arisen out of non-sport competitions such as TV shows, and have gradually acquired the emphasis on training, expertise, and professionalism that characterizes a sport, such as the various Ninja Warrior competitions.
Bottom line: the difference between any old game and a sport is largely in whether the right cultural practices have accreted around it. This is largely driven by the right sort of cultural acceptance: ways to select champions as proxies with which we can measure the physical or mental prowess of groups.
Adapted from an answer on Quora.