Here are a few parks in the neighborhood of the conference hotels:
Taddle Creek is probably the closest. This is where Gwyneth Paltrow would take her kids when she stayed either at the Intercontinental or the Four Seasons (which is now gone) during the Toronto International Film Festival. It has recently gone through a renovation and has some nice things for kids to do. (slide, swings, etc. mostly focused for younger kids and a water feature.) It's here.
If just a nice piece of grass is okay, there's Philosopher's walk which is pretty much across the street from the Intercontinental. It's got benches and trees and is a nice place, see here.
If you're willing to walk more, our favorite local park is Jean Sibelius Square. It's about a 10 minute walk from the hotels, but has been recently renovated and has good spaces for little kids and bigger kids. One of the added bonuses is that the space for the younger kids is "fenced" in which means that it doesn't require super vigilance. ;) There is also a nice grassy space and trees, benches and picnic tables. It's here.
Another park which is closer but is not the nicest of these parks since it has not been renovated in a long time is Huron-Washington Parkette. There is not a lot of nice places to sit when you're there with the kids, but does appeal to some. It's here.
Another park is the Huron Street Park, but this really doesn't have much grass or places to sit and is mostly for bigger kids. It's here.
If you do end up going in this direction, there are many places to go for lunch (see Matt's suggestions which are on the app and online) but here are two suggestions for ice cream: Greg's (on Bloor on west of Spadina) and Sweet Fantasies (398 Bloor St West). Greg's has interesting flavors like roasted marshmallow and Sweet Fantasies has a good spot to sit and people watch. If you want to get coffee as well, that's a bit trickier. You could go to the Green Beanery which is further west on Bloor, but I think that you should just get coffee at one of the places Matt suggests and get ice cream at a different place. (Apparently there is very little complementarity in the retailing of these two goods.)
For a green space that is bigger and nice is the Ramsden Park. It's further away and in the opposite direction, but there's a nice sandwich place near there and you could grab a sandwich and sit in the park. I don't think that there's much in the way of swings, etc., but it's just a nice green space. It’s here.
Finally, there's High Park. Going to this park will require taking the subway (9 stops from St. George stop which is the closest to the conference hotels), but it is the closest to Central Park in New York. There's a lot there with some nice trails, a great playground, and some places to get treats along with a restaurant. It's here.
One potentially important caveat regarding the parks: of all the local parks, the only one with a bathroom (and it's not a bad one) is Sibelius. This might be important for those with children who are in the midst of being potty-trained. Of course, there are bathrooms at Hyde Park and on the Island.
Some other ideas would be to go to the Royal Ontario Museum (ROM.on.ca) which is right across the street from the Intercontinental. It is a world cultures and natural history museum and is a pleasant place to spend time with kids. We used to go with our kids when the youngest was 2.5 years old and could spend several hours there. You could also go to the Harborfront, which is pleasant for a walk around or if you're more adventurous, take the ferry to Centre Island. (Take the subway to Union Station, walk to Queens Quay and take the ferry.) There is a cute little amusement park on the island and a lot of nice green space. Not sure about the playground facilities, but it's nice for a change of pace.
In addition to the ROM, there are two other museums that are very close to campus and the conference hotels. The first is the Bata Shoe museum. (You don't need to ask - I love it!) It's an okay place to wander around, though much of it will be of more interest to you than to the kids. (At least that's true with my kids.) The second is the Gardiner museum which houses Canada's national ceramics collection. It's sort of interesting and might be a good place to go if the weather isn't great.
Best, April