“…golf appeals to the idiot in us, and the child. … Just how childlike golf players become is proven by their frequent inability to count past five.”

http://www.imdb.com/name/nm0971329/bio

Source: Working Class Zero (2003), Chapter 19, p. 154

Here's Your Sign (1996)

"Twain probably never uttered [these] words," according to R. Kent Rasmussen, editor of The Quotable Mark Twain (1998).
"To play golf is to spoil an otherwise enjoyable walk" is found in H.S. Scrivener, "Memories of Men and Meetings" https://books.google.com/books?id=cYgCAAAAYAAJ&q=dicta#v=snippet&q=dicta&f=false, in Arthur Wallis Myers (ed.) Lawn Tennis at Home and Abroad New York:Charles Scribner’s Sons, 1903, p. 47. Scrivener attributes the aphorism to "my good friends the Allens". Reference from Quote Investigator http://quoteinvestigator.com/2010/05/28/golf-good-walk.
Misattributed

Though Woods might have used this phrase, it is one dating at least to the early 20th century; the earliest published use thus far located is in "Mr. Lyncargo's Professional, by Frank Savile, in ‪The Badminton Magazine of Sports and Pastimes‬ (1906), p. 498, where a character declares : "Life isn't all golf. There are other duties, sometimes."
Misattributed

How Do I Stay Normal in Hollywood (1942)

http://www.sportsfeelgoodstories.com/sport-quotes/sports-quotes/golf-quotes/10/21/2011.

Charlie Sifford hailed by Tiger Woods and others for breaking golf barriers https://www.pga.com/news/pga/charlie-sifford-hailed-tiger-woods-and-others-breaking-golf-barriers by The Associated Press (November 18, 2014)