It is strongly advised that anyone travelling with a child carries a consent letter signed by the non travelling parent(s).
A consent letter is not a legal requirement in Canada but it can simply travel as it may be requested by immigration officials. The letter confirms that the children have permission to travel abroad from their parents or guardians who are not travelling with them.
If you have a Court Order or a Separation Agreement, you should review the terms to see if there are any limitations on your ability to travel and if a consent letter is required.