Challenge coins mean different things to different troops. Senior enlisted and officers tend to place them on a desk to gloat to peers and the more junior troops slam them on the bar to see who's buying the next round. How you earn the coin also ranges widely, from "pleasure to meet you, have a coin" to "you made a great cup of coffee, have a coin" to even "you did something worthy of an award, but nah — have a coin."
Throughout history, warriors have carried coins, going as far back as the ancient Greeks and, eventually, the Romans. Coins carried by warriors were seen as the payment for the ferryman, Charon, to be exchanged for passage into the afterlife. They served as a memento mori, which roughly translates from Latin to mean, "remember that you will die." This is also the root of the English word 'memento.' Even back in the Civil War, troops from both armies carried coins with them as a cheap reminder of being back home.
The legend of the challenge coins, as we know them, started with WWI pilots. A young and rich lieutenant felt the need to flaunt his wealth to his new peers, so he spent his own money to buy solid bronze medallions of his unit insignia for his peers. Another pilot accepted it as a nice gift and wore it in a small leather pouch around his neck. Shortly after, he would be shot down behind enemy lines. He was captured by Germans, who stripped him over everything, but overlooked his medallion.
He escaped in civilian attire, crossed no man's land undetected, and stumbled into a French outpost. Unfortunately, the French didn't understand English nor his American accent and thought he was a German saboteur. The Germans took every bit of personal identification from the American pilot, so the only proof he had to show the French to not shoot him was the bronze unit medallion his rich peer gave him.
One French captor recognized the insignia and delayed his execution until they could confirm if he was American or not. Instead of a bullet to the head, the French gave him a bottle of wine and sent him on his way. When the pilot returned and told everyone of what happened, carrying those medallions became immediately important among all pilots. This also started the joke punishment of having to buy the next round if you're not carrying your coin.
Another lower enlisted tradition began in post-WWII Germany as assigned U.S. troops would carry West German money with them. The exchange rate was so bad that the One Pfennig coin was hardly worth a fraction of a penny and had nearly zero value to American troops. So, only the poorest of the poor would bother saving them — until troops gathered to drink. If someone would shout, "Pfennig check!" everyone would empty their pockets to see who was poor (if you had the near-worthless coin) and who wasn't (if you were above keeping them). If you were "rich" enough to not need to carry a worthless coin, you were rich enough to buy your brother-in-arms a drink. This soon shifted to include challenge coins, which also had no monetary value.