I just spoke to someone in my family who works in the social work/child protection area and they advised it is concerning and advised the following:
1. Keep talking to your son, praise him for being open with you and keep encouraging it. Ask him how he felt about, ask him if it made him feel weird (use that phrase) and try and get his feelings on it. Talk to your older son and see if anything else has been said.
2. If possible pull him out of the program, these workers know they are not supposed to suggest kids coming back to their homes or anything similar. It's not just a rough guide it's a very firm order.
3. Request a meeting with the center management and that worker, ask for an explanation, ask the worker at the meeting to show his working with children check and then expect some actions, i.e. additional training for the worker, reprimanded etc.
4. Above all make sure your son feels comfortable talking with you about it, sounds like you're already on top of this but keep it going, gentle and encouraging.
There are most likely three explanations for this guys behaviour:
1. He's an idiot. Some people just make stupid remarks or mistakes, it's a definite no-no in these jobs but it does happen.
2. He just does what he wants, some people enter the teaching type professions thinking it's Dead Poets Society and they can do what they want because they want to teach/look after people in a certain way. Doesn't mean he's abusive just thinks he can buck the rules.
3. He's a paedophile. Unfortunately people like this tend to be attracted to areas where they work with children, it can be a an almost all-consuming parts of their lives and they adjust their lives to suit it, hence their job choices. Unfortunately they're not always old men in dirty coats, they can also be young people who look cool and whom younger children are attracted to, they will also use things that are attractive to kids to attract and groom them.
I'm not saying that this guy is a paed. but it is a potential risk (people shouldn't start panicking and lynching random dodgy strangers but parents need to be on the ball with this as they tend to go where kids are - my relative has done this for a long time and the recent royal commission also highlights this.) Don't bother calling the police first up, they won't do anything based on that, best thing you can do is remove your children, talk to center management and see the response from them. This behavior is wrong and if they should act accordingly (I can't stress this enough, officially it's a really big no-no), if they appear to be protecting their worker you should then take it to the next level of management (i.e. if it's a council program take it to the council and demand a response in 24 hours not 3 weeks), you may also at this stage want to consider the police, they probably won't do anything with that level of evidence but if this individual has been doing this before or continues this will build a picture and hopefully some usable evidence.
I hope this helps, it sounds like you're doing the right thing and it's also good that your other son was there as well as he sounds like he's been keeping an eye on your younger son.
Cheers
Zarniwoop