What is the youngest age child you would take backpacking in the Grand Canyon?

I've backpacked the Grand Canyon several times. I'm considering taking my kids to Indian Gardens this summer. Indian Gardens is in the canyon, and the trail is 4.6 miles. My kids are 6 and 4 and 2. My 2-year-old would ride in a child carrier, of course. My 6-year-old (almost 7) is a phenomenal hiker, so I'm not worried about him at all. My 4 year old (almost 5) is just starting to do "real" hikes this year. He's done 8.5 miles here in the Colorado Rockies, but the terrain wasn't as steep as what we'd encounter in the G.C. I believe he could do it just fine, if we took our time. Am I crazy, though? Does anyone ever take kids that young below the rim? I don't want to do the hike as a dayhike - I definitely know the 4-year-old couldn't do all that in one day.